:W&Bá¾,hl>M c0myqu37ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0ÍBŒ%ú‚" g9a4084dogmc c0myqu37ŠÖ c0myqu37juju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.20887652657237368 epochs‚10  hidden_dim‚128 & learning_rate‚0.000731799474931662  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚6Bjjtttt_blocks_c0myqu37bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvŠ ÇðêÔ°¹¯ûš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64hZƒ!7 c0myqu37&  _runtime‚0  _wandb.runtime‚0„eÅjÒä -Linux-6.8.0-137-generic-x86_64-with-glibc2.36CPython 3.12.10 ÆðêÔ¨ÙÐö*-of*=/data/spatop/options_files/sweep_tttt_jj/config_c0myqu37.json*-l* /scratch/logs*-n*jjtttt_blocks_c0myqu372-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvr/opt/conda/bin/pythonº dr7331jhq9ad9lpshtdyd30n7ltqorix¡*¾ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€øáo€À¦µçTª€àäêÞÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-c1fdb213-5a16-84e3-13b4-cf29bc35dbf8Ê13.2º dr7331jhq9ad9lpshtdyd30n7ltqorixµª½Ç2  requirements.txt µy-W c0myqu37ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64» ¾Y c0myqu37ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ý9¼*  c0myqu37b trainer/global_step2¢ëÙ#\  c0myqu37ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64UsØ/  c0myqu37b*"trainer/global_step2÷yºåK  c0myqu37j: ÈðêÔ˜ßòŽ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ò™Öå’  c0myqu37j€  ÉðêÔèÚ¦uî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 53.9 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¾u[ c0myqu37jÉ ÉðêԀФv·Trainable params: 56.7 M Non-trainable params: 86 Total params: 56.7 M Total estimated model params size (MB): 226.908 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 œ'mp c0myqu37j_ ÉðêÔˆ ÁvPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ¥ö½£z c0myqu37ji ÉðêÔØ‘ËvZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). X«[Òx c0myqu37jg ÉðêÔ˜µÔvXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ê°ã c0myqu37jn ÉðêÔ¨ëÜv_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). éK¹y c0myqu37jh ÉðêÔ€ŠåvYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). võs c0myqu37jb ÉðêÔøÇívSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ›¯è­p c0myqu37j_ ÉðêÔ˜ÏõvPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ñvs c0myqu37jb ÉðêÔ¸ÖývSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). i 4}m c0myqu37j\ ÉðêÔ¨í…wMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). l‰Ü«l c0myqu37j[ ÉðêÔ°üwLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). X‡ËÜ c0myqu37*Ê  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"zèIßê c0myqu37jØ ÉðêÔðäë}È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ›ÀhM c0myqu37j» ÉðêÔЉÿ}«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Œ… ë c0myqu37jÙ ÌðêÔðûœöÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. Ä‘æP c0myqu37j¾ ÌðêÔø­Ãö­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. `Ð µ“‚(:‰ ×ðêÔׇõÞ gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'óëï‚(:å ×ðêÔ°Ô£ä disk.sdb.in‚0 disk.sdb2.out‚ 2.1357421875 disk.nvme0n1.out‚0.65625 disk.sda.out‚ 2.1357421875 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 287.15625 disk.sda1.in‚0 disk.sdb2.in‚0 cpu‚24.442726853615003 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚ 0.01171875) proc.memory.percent‚17.57338047027588 network.recv‚41777) disk./.usagePercent‚75.86575777956322 disk.nvme0n1.in‚0 disk.md0.out‚ 2.0546875$ memory_percent‚91.34600162506104 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚ 2.1357421875 disk.md0.in‚0 disk.sdb.out‚ 2.1357421875 disk.sda.in‚0) proc.memory.availableMB‚ 1417.87109375 network.sent‚33634$ disk./.usageGB‚2713.326972961426 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.md0p1.out‚ 2.0546875 disk.sdb1.out‚0 disk.rbd2.out‚ 0.07421875 disk.dm-0.in‚0# proc.memory.rssMB‚ 2879.22265625 disk.dm-0.out‚287.8125Iè““ ‚(:‰ æðêÔĨ˜ß gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÈ^6ø!‚(:î æðêÔ¨ŠÊà disk.sda2.in‚0 disk.dm-0.out‚ 541.44140625 disk.sdb2.in‚0 disk.sda.out‚ 3.6416015625 disk.md0p1.in‚0% disk./.usageGB‚2713.3266639709473 disk.sda.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.07421875 disk.sda1.in‚0 disk.rbd2.out‚ 0.07421875 disk.sda2.out‚ 3.6416015625$ memory_percent‚91.35406017303467 disk.rbd2.in‚0 network.sent‚46744 disk.sdb.out‚ 3.6416015625 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 540.58203125 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 3.5078125" proc.memory.rssMB‚ 2879.3359375 disk.dm-0.in‚0 disk.sdb2.out‚ 3.6416015625 disk.rbd0.in‚0 network.recv‚52754) disk./.usagePercent‚75.86574914006012 disk.sdb.in‚0 disk.nvme0n1.out‚0.859375 cpu‚24.591880370176114 disk.md0p1.out‚ 3.5078125 disk.sda1.out‚0) proc.memory.availableMB‚ 1416.55078125* proc.memory.percent‚17.574071884155273ß^Çy“"‚(:‰ õðêÔ”»êÞ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äÿ2ˆø#‚(:î õðêÔÓÏ«à* proc.memory.percent‚17.574167251586914 disk.sdb.in‚0 disk.sdb.out‚ 5.17041015625 disk.nvme1n1.out‚ 692.609375 disk.sdb1.out‚0 disk.sda1.out‚0 network.recv‚78474 disk.dm-0.out‚ 694.1015625 cpu‚24.677209797126654 proc.cpu.threads‚17% disk./.usageGB‚2713.3520545959473 disk.rbd2.out‚ 0.1533203125 disk.nvme0n1.in‚0 disk.md0.out‚4.96875$ memory_percent‚91.36154651641846 network.sent‚63596) disk./.usagePercent‚75.86645907256194 disk.sdb2.out‚ 5.17041015625 disk.sda.in‚0 disk.md0p1.in‚0 disk.sda2.out‚ 5.17041015625 disk.md0.in‚0 disk.sda.out‚ 5.17041015625 disk.rbd0.in‚0 disk.nvme0n1.out‚ 1.4921875 disk.sda2.in‚0 disk.sdb2.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.07421875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sdb1.in‚0" proc.memory.rssMB‚ 2879.3515625 disk.md0p1.out‚4.96875) proc.memory.availableMB‚ 1415.324218753c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPñœm-&) _wandbruntime‚57  _runtime‚57p¨Â'½ % _timestamp‚1.7885246736558611e+09 _runtime‚ 57.191551418 _step‚0 ' lr-AdamW/pg1‚2.9659366643218727e-05 ' lr-AdamW/pg2‚2.9659366643218727e-05 trainer/global_step‚49Fkb[Ä(¿ ' lr-AdamW/pg2‚2.9659366643218727e-05 trainer/global_step‚49 % _timestamp‚1.7885246736558611e+09 _runtime‚ 57.191551418 _step‚0 ' lr-AdamW/pg1‚2.9659366643218727e-05ô´l)bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPî·È_s*bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÞóŠxi+be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPû~³`,b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPöC èi-be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP)êÀk.bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPW(šn/bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPŒL@7p0bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP%æÕm1bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP…0"àg2bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP\Ï\rf3bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPþ ¼Îq4bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP;FNïp5bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPPq]Á^6bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPJM+Ìf7bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPþB¨DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPM<Þžobk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPW “x?bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPÀ³ÐŒc@b_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP™·µëlNbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP´íåmObi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP¸¿þEtPbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPçw%ßiQbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÅVènRbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP˜Çù,fSbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPz-A:iTbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPSs݈qUbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP²GéiVbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP¿wâkWbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPy_nXbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP NXkYbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPA#¸qZbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP‹"V©c[b_ qbbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP¥@¦’gcbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP?éÒEpdbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPðè~:tebp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPú®zWfbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP¦êçkgbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP gm—bhb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP³ì1_ib[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPx;4Îsjbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPhÂikbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP‡.×Relba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPg²emba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÓIµsgnbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPyänobj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPclQ«fpbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP9˜‚jqbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPn“¿Wfrbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP”‘Vmsbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPIèôõitbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP4ø€Sjubf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP·wyúlvbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPùˆ'Œgwbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÕv´—hxbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP+šayb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP‘ù!2pzbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP³…–ô_{b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP./ep|bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP¨Xžj`}b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP—ìKzr~bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPô½Éïpbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPw 8o€bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPp®Þ•wbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPuˆh‚bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPn‡tƒbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPV¾½èh„bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP¤>¤n…bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP„œ°Àj†be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP äu`‡b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP>¼ ÷pˆbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP!Éìk‰bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP7®kŠbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPÿiëDi‹bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPÓj1yŒbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPæ&XYpbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP—ì§ooŽbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÖð áobj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÿW€Ãeb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP$"Òn‘bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPЩ2Å^’bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP~Åði“bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPs1FŒe”b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP9]Îo•bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP·öÄs–bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPôü´{k—bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPhoL_˜bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPÄÎK­k™bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP7ñ1tšbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÿ2œad›b_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP>¦Ê­m¡bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPdîi‡m¢bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¿ÝÜis£bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP8 ß.¤) _wandbruntime‚57  _runtime‚57”5Àh¥šO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189000192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4591679035392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1235123 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚289646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4532275660288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4161441198080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_ÑA¥žbytes.all.peak‚ 14134970368 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚610298 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4104811093504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56418114260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018792 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1259969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59379435008 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 55510146804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4028224489828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134970368 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4104811093504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4029149496644 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 _runtime‚ 57.192519772 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152ãJ„¤'¦žO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4104811093504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4029149496644 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 _runtime‚ 57.192519772 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189000192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4591679035392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1235123 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚289646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4532275660288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4161441198080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134970368 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚610298 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4104811093504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56418114260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018792 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1259969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59379435008 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 55510146804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4028224489828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134970368 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300Bd|†`§b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP°Ò¤¤c¨b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPgÞ¶Am©bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPsðñÜVªbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP9€ö°f«ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP„¿G]¬bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP2ùAÙg­bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPù:º»d®b_ a°b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPå‹Séo±bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPSø:fk²bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP<=Žì]³bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP’”ªp´bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP>î"!mµbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP;‹úÆb¶b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPø´(Ép·bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP÷!Om¸bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP˜!°d¹b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPåö„eÃb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPªÔ @qÄbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP Ì™geÅb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP SoÆbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPB÷íGoÇbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPX–GhÈbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPˆlÖbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPNv•e×b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPD¥ŽqØbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPŸ¦sÜhÙbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP†3ñÀiÚbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP²Êo8iÛbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPHý»jÜbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPEªÐŸjÝbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP±$8mnÞbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP#ùåWdßb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPF®áFmùbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÂzí½uúbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP_AÓ5kûbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP7ƒkübf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP/Ávkýbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPáÿVjþbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPåŒzÞgÿbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP™9zp€bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP¼èÑgbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP¶tq‚bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPª©O(bƒb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP~² nd„b_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP³FÒõi†bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP™V–r‡bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP 7´[eˆb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP«km‰bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPl‡oeŠb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPÞ ßDh‹bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPBÈ jŒbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPŸÖj#wbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPWìÍ\mŽbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÔûåÁlbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPQ6‚6hbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP_ à^‘bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPÃÈFf’ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP½~D b“b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPOU%Oh”bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP®¸ã³_•bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPš|ÐÉm–bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPÀ¸—Df—ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP6bYùf˜ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPqŠýAm™bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP%^ª{gšbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPegªp›bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP‘û€Ëpœbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPÊxæŠnbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP©B>hžbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP¨Ö‡`rŸbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP3d`^ bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP;gš‰f¡ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPM"¸.¢) _wandbruntime‚58  _runtime‚58Æ+Õ`£¢M aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4622525304832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 MGDeviceStatsMonitor.o‹Èë³Hn_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 _step‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚ 58.1674015 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4051731937568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4051748976928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4243254536704 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362kl!¬&¤¦M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4622525304832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 _step‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚ 58.1674015 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4051731937568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4051748976928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4243254536704 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569IßyC¥b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPìžA¦b< loss/FBt4/assignment_loss"trainer/global_step2:JP¶šÆ@§b; loss/FRt1/detection_loss"trainer/global_step2:JP´£>Ï@¨b; loss/FRt2/detection_loss"trainer/global_step2:JP{æÕ@©b; loss/FRt3/detection_loss"trainer/global_step2:JP÷w¬{@ªb; loss/FBt2/detection_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JPi$ƒqA¬b< loss/FRt3/assignment_loss"trainer/global_step2:JPu,¯‰-­b( epoch"trainer/global_step2:JP¡L‹gA®b< loss/FRt1/assignment_loss"trainer/global_step2:JP‹×A¯b< loss/FBt3/assignment_loss"trainer/global_step2:JP+ã8B°b= loss/SRqqt1/detection_loss"trainer/global_step2:JP!È}B±b= loss/SRqqt3/detection_loss"trainer/global_step2:JPÀº@²b; loss/FBt3/detection_loss"trainer/global_step2:JPf$|A³b< loss/FRt4/assignment_loss"trainer/global_step2:JP @B´b= loss/SRqqt2/detection_loss"trainer/global_step2:JP>gFAµb< loss/FRt2/assignment_loss"trainer/global_step2:JPqA¶b< loss/FBt1/assignment_loss"trainer/global_step2:JP#=ÓB·b= loss/SRqqt4/detection_loss"trainer/global_step2:JP ò™D@¸b; loss/FBt4/detection_loss"trainer/global_step2:JPÑhy1C¹b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPì÷ûÕAºb< loss/FBt2/assignment_loss"trainer/global_step2:JPXÔçj@»b; loss/FBt1/detection_loss"trainer/global_step2:JP¡[eÓC¼b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP®È4C½b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPÒ‹·i@¾b; loss/FRt4/detection_loss"trainer/global_step2:JPyUÝ.¿) _wandbruntime‚58  _runtime‚58Š=³÷EÀ¿ 2loss/SRqqt3/assignment_loss‚5.4811598539352415 /loss/FBt4/assignment_loss‚2.734989300850899 .loss/FRt1/detection_loss‚4.200475841760635 .loss/FRt2/detection_loss‚4.692223537832706 /loss/FRt3/detection_loss‚2.9726953801923783 ,loss/FBt2/detection_loss‚15.654006941752 &loss/total_loss‚145.77178625307207 trainer/global_step‚49 0loss/FRt3/assignment_loss‚7.3383867493991195 epoch‚0 _runtime‚ 58.169069268 /loss/FRt1/assignment_loss‚7.363090813159943 /loss/FBt3/assignment_loss‚2.753983039122361 0loss/SRqqt1/detection_loss‚5.250696329632774 1loss/SRqqt3/detection_loss‚2.7143037686745326 /loss/FBt3/detection_loss‚13.330862200030914 /loss/FRt4/assignment_loss‚7.163748264312744 0loss/SRqqt2/detection_loss‚5.364225371275097 /loss/FRt2/assignment_loss‚7.023475513458252 .loss/FBt1/assignment_loss‚2.67416250705719 1loss/SRqqt4/detection_loss‚2.0244818296231015 .loss/FBt4/detection_loss‚1.820899957575039 # _timestamp‚1.78852467463377e+09 1loss/SRqqt1/assignment_loss‚5.496267139911652 0loss/FBt2/assignment_loss‚2.7444079355760054 /loss/FBt1/detection_loss‚23.638071198016405 1loss/SRqqt2/assignment_loss‚5.527548611164093 1loss/SRqqt4/assignment_loss‚5.310883313837186 .loss/FRt4/detection_loss‚2.496740854921795 _step‚3±¼U,IÁà /loss/FRt2/assignment_loss‚7.023475513458252 .loss/FBt1/assignment_loss‚2.67416250705719 1loss/SRqqt4/detection_loss‚2.0244818296231015 .loss/FBt4/detection_loss‚1.820899957575039 # _timestamp‚1.78852467463377e+09 1loss/SRqqt1/assignment_loss‚5.496267139911652 0loss/FBt2/assignment_loss‚2.7444079355760054 /loss/FBt1/detection_loss‚23.638071198016405 1loss/SRqqt2/assignment_loss‚5.527548611164093 1loss/SRqqt4/assignment_loss‚5.310883313837186 .loss/FRt4/detection_loss‚2.496740854921795 _step‚3 2loss/SRqqt3/assignment_loss‚5.4811598539352415 /loss/FBt4/assignment_loss‚2.734989300850899 .loss/FRt1/detection_loss‚4.200475841760635 .loss/FRt2/detection_loss‚4.692223537832706 /loss/FRt3/detection_loss‚2.9726953801923783 ,loss/FBt2/detection_loss‚15.654006941752 &loss/total_loss‚145.77178625307207 trainer/global_step‚49 0loss/FRt3/assignment_loss‚7.3383867493991195 epoch‚0 _runtime‚ 58.169069268 /loss/FRt1/assignment_loss‚7.363090813159943 /loss/FBt3/assignment_loss‚2.753983039122361 0loss/SRqqt1/detection_loss‚5.250696329632774 1loss/SRqqt3/detection_loss‚2.7143037686745326 /loss/FBt3/detection_loss‚13.330862200030914 /loss/FRt4/assignment_loss‚7.163748264312744 0loss/SRqqt2/detection_loss‚5.364225371275097thý“‚(:ˆ „ñêÔÉÂß gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.718' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.6795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýƒÃ‚(:õ „ñêÔÈÕÂà proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.md0.in‚0 disk.md0.out‚6.390625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.1533203125* proc.memory.percent‚17.574334144592285) disk./.usagePercent‚75.86827070170001 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚6.390625 disk.sdb.out‚ 6.64697265625 disk.sda.in‚0 disk.rbd0.out‚ 0.1533203125# proc.memory.rssMB‚ 2879.37890625 network.sent‚83223 disk.sda2.out‚ 6.64697265625 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ memory_percent‚91.37918949127197$ disk./.usageGB‚2713.416847229004 disk.sdb1.out‚0) proc.memory.availableMB‚ 1412.43359375 cpu‚24.73476998922001! disk.nvme1n1.out‚ 850.92578125 disk.sda1.in‚0 network.recv‚108215 disk.dm-0.out‚ 852.80859375 disk.sdb.in‚0 disk.sdb2.out‚ 6.64697265625 disk.nvme0n1.out‚ 1.8828125 disk.sda.out‚ 6.64697265625v‚áf”Ä‚(:‰ “ñêÔÚþÂà gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09»| üÅ‚(:ñ “ñêÔû¢…á disk.sdb.out‚ 8.62939453125 disk.sda1.in‚0 disk.nvme0n1.out‚ 1.8828125* proc.memory.percent‚17.574381828308105# proc.memory.rssMB‚ 2879.38671875 cpu‚24.774693084040013 disk.sdb2.out‚ 8.62939453125 disk.dm-0.in‚0 disk.sda.in‚0 disk.md0p1.out‚8.3125 disk.rbd2.in‚0$ memory_percent‚91.37725830078125 network.sent‚139421) disk./.usagePercent‚75.87352266630168 disk.sda2.in‚0 disk.md0p1.in‚0 disk.rbd0.out‚ 0.1533203125 disk.rbd2.out‚ 0.21630859375# proc.memory.availableMB‚1412.75 disk.sdb2.in‚0% disk./.usageGB‚2713.6046829223633 disk.md0.out‚8.3125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 1060.07421875 disk.rbd0.in‚0 disk.nvme0n1.in‚0 network.recv‚117714 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 1061.95703125 disk.sda.out‚ 8.62939453125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚ 8.62939453125 disk.md0.in‚0 disk.sdb.in‚09Ne”Æ‚(:‰ ¢ñêÔËϧà gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.579' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.89475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ð Ç‚(:þ ¢ñêÔð€Ñà$ memory_percent‚91.37718677520752) proc.memory.availableMB‚ 1412.76171875# proc.memory.rssMB‚ 2879.38671875 disk.sdb.in‚0 disk.sdb.out‚10.18408203125 disk.rbd0.in‚0 disk.sdb1.out‚0* proc.memory.percent‚17.574381828308105 disk.sda.in‚0 disk.rbd0.out‚ 0.21630859375 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 1.8828125 disk.sda2.in‚0 disk.md0.in‚0 disk.md0p1.out‚ 9.81640625 proc.cpu.threads‚17 disk.md0.out‚ 9.81640625 cpu‚24.80672507404183 network.recv‚122937% disk./.usageGB‚2713.8551864624023 disk.nvme1n1.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚10.18408203125 disk.nvme1n1.out‚ 1451.4375 disk.dm-0.in‚0 disk.dm-0.out‚ 1453.3203125) disk./.usagePercent‚75.88052685012505 disk.sda2.out‚10.18408203125 disk.sda.out‚10.18408203125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚146142 disk.md0p1.in‚0 disk.rbd2.out‚ 0.21630859375¨i§”È‚(:‰ ±ñêÔ€Üà gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï#WûÉ‚(:ù ±ñêÔ“Çöà disk.rbd0.out‚ 0.21630859375 cpu‚24.829089313234903 proc.cpu.threads‚17 disk.sda2.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.md0p1.in‚0 network.sent‚154753% disk./.usageGB‚2713.8541679382324 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.md0p1.out‚ 11.3515625( proc.memory.availableMB‚ 1412.7265625 disk.sda.out‚ 11.775390625" disk.nvme1n1.out‚ 1686.42578125) disk./.usagePercent‚75.88049837176298 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.574381828308105 disk.sda.in‚0 network.recv‚127938 disk.rbd2.in‚0 disk.nvme0n1.out‚ 1.98828125 disk.sda2.out‚ 11.775390625 disk.md0.out‚ 11.3515625 disk.sdb.in‚0 disk.sdb2.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 1688.4140625 disk.sdb.out‚ 11.775390625 disk.sdb2.out‚ 11.775390625$ memory_percent‚91.37740135192871# proc.memory.rssMB‚ 2879.38671875`À»k0Ê+ _wandbruntime‚110  _runtime‚110•¢”Â˼ % _timestamp‚1.7885247267924254e+09 _runtime‚ 110.328074811 _step‚4 & lr-AdamW/pg1‚5.992402648323783e-05 & lr-AdamW/pg2‚5.992402648323783e-05 trainer/global_step‚99µ-@IÆÌÀ & lr-AdamW/pg1‚5.992402648323783e-05 & lr-AdamW/pg2‚5.992402648323783e-05 trainer/global_step‚99 % _timestamp‚1.7885247267924254e+09 _runtime‚ 110.328074811 _step‚4ö¥®0Í+ _wandbruntime‚110  _runtime‚110_Šh\ΧO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2480025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4531665 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 112295369368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 110.329113736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026794 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8185765384704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7923594070224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 _step‚5 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8297555706880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112717529600 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9161275520000 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8298482914304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111808325120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8185765384704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112717529600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1791489 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8185747381760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1211 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1213550 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 D8DeviceS²ýX@QtatsMonitor.on_train_batch_start/active.all.freed‚4506817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111387401912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 % _timestamp‚1.7885247267933607e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9161015739392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚99 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9043584822784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8298482914304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7923611109584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚577335 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8035906478952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9043560882688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 117690697216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2504871 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111808325120 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112717529600 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9161275520000 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8298482914304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111808325120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8185765384704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112717529600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1791489 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8185747381760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1211 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1213550 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4506817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111387401912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 % _timestamp‚1.7885247267933607e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9161015739392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚99 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9043584822784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712Û˜£Å0Ð+ _wandbruntime‚111  _runtime‚111ÛìŸJ·&ѱM MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2504856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2046952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚583083 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 118635048448 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112929785344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1211 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113412642854 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2529728 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8002628704128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225613 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚ 111.437753022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8267366467584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8267384470528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8002611664768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885247279012308e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113839003136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8115116339958 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 _step‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2504856 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8381223473664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚99 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9133786587136 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8381223473664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8267384470528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113412642854 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2529728 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8002628704128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225613 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚ 111.437753022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8267366467584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8267384470528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8002611664768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885247279012308e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113839003136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor¶ \Ç.on_train_batch_end/segment.small_pool.allocated‚569 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8115116339958 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 _step‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2504856 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8381223473664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚99 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9133786587136t…(~0Ó+ _wandbruntime‚111  _runtime‚111#! JÔÄ .loss/FRt4/detection_loss‚2.130820786274636 /loss/FBt4/detection_loss‚2.9787058022299107 epoch‚0 0loss/FBt1/assignment_loss‚1.5507459860098989 0loss/FBt4/assignment_loss‚1.5533356634346214 2loss/SRqqt4/assignment_loss‚5.2917952390817495 0loss/SRqqt3/detection_loss‚2.689712064119017 /loss/FRt1/assignment_loss‚7.327781013820482 /loss/FRt4/assignment_loss‚7.147992468591946 trainer/global_step‚99 /loss/FRt3/assignment_loss‚7.332429769304063 -loss/FRt2/detection_loss‚4.66346796353658 /loss/FRt3/detection_loss‚3.8830493807792665 0loss/SRqqt1/detection_loss‚5.047017233336673 0loss/SRqqt2/detection_loss‚3.883150404149836 $ _timestamp‚1.788524727903549e+09 .loss/FRt1/detection_loss‚3.616768028425134 0loss/FBt2/assignment_loss‚1.5743104740977287 /loss/FBt3/assignment_loss‚1.481879886239767 .loss/FBt1/detection_loss‚4.559945909992645 /loss/FBt3/detection_loss‚10.747681491076946 &loss/total_loss‚113.80152092036343 /loss/FRt2/assignment_loss‚7.330192283347801 1loss/SRqqt2/assignment_loss‚5.429089762947776 _runtime‚ 111.439481229 _step‚7 1loss/SRqqt1/assignment_loss‚5.421206796870512 1loss/SRqqt3/assignment_loss‚5.472877658143336 1loss/SRqqt4/detection_loss‚2.1605748953727577 /loss/FBt2/detection_loss‚10.526989959180355Ush”NÕÈ /loss/FRt3/detection_loss‚3.8830493807792665 0loss/SRqqt1/detection_loss‚5.047017233336673 0loss/SRqqt2/detection_loss‚3.883150404149836 $ _timestamp‚1.788524727903549e+09 .loss/FRt1/detection_loss‚3.616768028425134 0loss/FBt2/assignment_loss‚1.5743104740977287 /loss/FBt3/assignment_loss‚1.481879886239767 .loss/FBt1/detection_loss‚4.559945909992645 /loss/FBt3/detection_loss‚10.747681491076946 &loss/total_loss‚113.80152092036343 /loss/FRt2/assignment_loss‚7.330192283347801 1loss/SRqqt2/assignment_loss‚5.429089762947776 _runtime‚ 111.439481229 _step‚7 1loss/SRqqt1/assignment_loss‚5.421206796870512 1loss/SRqqt3/assignment_loss‚5.472877658143336 1loss/SRqqt4/detection_loss‚2.1605748953727577 /loss/FBt2/detection_loss‚10.526989959180355 .loss/FRt4/detection_loss‚2.130820786274636 /loss/FBt4/detection_loss‚2.9787058022299107 epoch‚0 0loss/FBt1/assignment_loss‚1.5507459860098989 0loss/FBt4/assignment_loss‚1.5533356634346214 2loss/SRqqt4/assignment_loss‚5.2917952390817495 0loss/SRqqt3/detection_loss‚2.689712064119017 /loss/FRt1/assignment_loss‚7.327781013820482 /loss/FRt4/assignment_loss‚7.147992468591946 trainer/global_step‚99 /loss/FRt3/assignment_loss‚7.332429769304063 -loss/FRt2/detection_loss‚4.66346796353658`yŽ&“Ö‚(:ˆ ÀñêÔÐÉéà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðü3ׂ(:ø ÀñêÔíèƒá disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sdb.in‚0) proc.memory.percent‚17.57504940032959 network.sent‚159224 disk.sdb2.out‚ 13.7783203125 disk.nvme1n1.out‚ 1999.546875 disk.sdb1.in‚0 disk.dm-0.out‚ 2001.53515625$ memory_percent‚91.37873649597168 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.sdb2.in‚0 disk.sda.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.out‚ 13.2890625 disk.sdb.out‚ 13.7783203125( proc.memory.availableMB‚ 1412.5078125 network.recv‚132983 disk.sda.out‚ 13.7783203125 disk.nvme1n1.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2879.49609375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 1.98828125 disk.sda2.out‚ 13.7783203125 proc.cpu.threads‚17( disk./.usagePercent‚75.8805000783315% disk./.usageGB‚2713.8542289733887 disk.md0p1.in‚0 disk.sda2.in‚0 cpu‚24.850333738182666 disk.md0p1.out‚ 13.2890625Èd˜ ’Ø‚(:‡ ÏñêÔòÖ­à gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rñ%Ù‚(:ø ÏñêÔšú“á network.sent‚208103 network.recv‚138489 disk.dm-0.in‚0 disk.sda.out‚15.21142578125 disk.md0p1.out‚14.65625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.57504940032959 disk.md0.in‚0 disk.sdb2.out‚15.21142578125 disk.rbd2.in‚0$ memory_percent‚91.37904644012451 disk.sdb.out‚15.21142578125 disk.sdb2.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚15.21142578125 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875$ disk./.usageGB‚2713.327060699463 disk.sdb.in‚0) proc.memory.availableMB‚ 1412.45703125 cpu‚24.865988461729163) disk./.usagePercent‚75.86576023275546 disk.md0.out‚14.65625 disk.rbd0.out‚ 0.279296875 disk.sda2.in‚0 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.nvme0n1.out‚ 2.19921875# proc.memory.rssMB‚ 2879.49609375 disk.dm-0.out‚ 2131.34375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 2129.14453125 disk.nvme0n1.in‚0®H|Õ”Ú‚(:‰ ÞñêÔ¬ô¹à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚153.509' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.37725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!´«v Û‚(:ÿ ÞñêÔúÌûá disk.sda2.in‚0 disk.dm-0.out‚ 2437.58203125 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 0.279296875( proc.memory.availableMB‚ 1412.4609375 network.recv‚142967 disk.sda2.out‚16.87353515625 disk.md0.in‚0! disk.nvme1n1.out‚ 2435.3828125 disk.rbd0.out‚ 0.279296875# proc.memory.rssMB‚ 2879.49609375 cpu‚24.881615895534765 disk.sdb.out‚16.87353515625 disk.sdb2.in‚0 disk.sdb2.out‚16.87353515625 disk.sda.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.57504940032959% disk./.usageGB‚2713.2821655273438 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 2.19921875 disk.sda1.out‚0 disk.sdb.in‚0 disk.md0p1.out‚ 16.24609375 disk.sda.out‚16.87353515625# memory_percent‚91.3790225982666 network.sent‚216386 disk.md0.out‚ 16.24609375 disk.dm-0.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0) disk./.usagePercent‚75.86450494495293ߨޔ܂(:‰ íñêÔÃïŸß gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v'U­ýÝ‚(:ò íñêÔ´™îà) disk./.usagePercent‚75.86646248569896 disk.rbd2.in‚0 disk.md0p1.in‚0 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.out‚ 74.046875( proc.memory.percent‚17.5750732421875 disk.sda.out‚18.83935546875 disk.sda2.out‚18.83935546875% disk./.usageGB‚2713.3521766662598 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 2567.6328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 0.279296875 network.recv‚148112 cpu‚24.89264242973206 disk.dm-0.in‚0 disk.dm-0.out‚ 2641.6796875 disk.sdb2.in‚0 disk.nvme0n1.in‚0 proc.memory.rssMB‚2879.5 proc.cpu.threads‚17 disk.sdb.in‚0 disk.sdb.out‚18.83935546875 disk.sdb2.out‚18.83935546875 disk.md0p1.out‚ 18.15234375 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚91.37916564941406 network.sent‚220918 disk.md0.in‚0 disk.md0.out‚ 18.15234375 disk.sda.in‚0% proc.memory.availableMB‚ 1412.4375Á îÔ0Þ+ _wandbruntime‚165  _runtime‚165·r5Âß¼ & lr-AdamW/pg2‚9.018868632325695e-05 trainer/global_step‚149 $ _timestamp‚1.788524782298388e+09 _runtime‚ 165.833987892 _step‚8 & lr-AdamW/pg1‚9.018868632325695e-05Fç%vÆàÀ & lr-AdamW/pg1‚9.018868632325695e-05 & lr-AdamW/pg2‚9.018868632325695e-05 trainer/global_step‚149 $ _timestamp‚1.788524782298388e+09 _runtime‚ 165.833987892 _step‚8.‘´0á+ _wandbruntime‚165  _runtime‚165È M¾'â¸O L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175721459712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167264657020 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3034792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2683490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167895750144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11874490836784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 % _timestamp‚1.7885247822995787e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3034794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 trainer/global_step‚149 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 168172624476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _step‚9 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13554870045184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798£1Â'ã¼O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175721459712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167264657020 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3034792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2683490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167895750144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11874490836784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 % _timestamp‚1.7885247822995787e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3034794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 trainer/global_step‚149 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 168172624476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _step‚9 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13554870045184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 f¢0ä+ _wandbruntime‚166  _runtime‚166wÍ¥óåÃM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 _runtime‚ 166.965566868 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1828863 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12518265189888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12348338761728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚871772 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3774630 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 169926428160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytqPÖes.small_pool.allocated‚ 1193279488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12348338761728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚872390 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 169017210368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3749758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12122798329290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 177123267584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11953508431328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12518265189888 % _timestamp‚1.7885247834293408e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3749758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3054952 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3054954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚10 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 169289897962 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1828865 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2701255 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615Æ'ÙeÍ&æÇM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 169926428160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12348338761728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚872390 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 169017210368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3749758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12122798329290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 177123267584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11953508431328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12518265189888 % _timestamp‚1.7885247834293408e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3749758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3054952 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3054954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚10 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 169289897962 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1828865 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2701255 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 _runtime‚ 166.965566868 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1828863 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12518265189888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12348338761728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚871772 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3774630 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 "{Œ0ç+ _wandbruntime‚166  _runtime‚166P%QèË /loss/FRt1/assignment_loss‚7.303622334502464 1loss/SRqqt1/assignment_loss‚3.731844698941266 2loss/SRqqt4/assignment_loss‚3.3092575719801047 0loss/SRqqt2/detection_loss‚2.619613775762461 1loss/SRqqt4/detection_loss‚2.1295591809992063 2loss/SRqqt2/assignment_loss‚3.9653797392117776 0loss/FBt2/assignment_loss‚0.6506337042083032 .loss/FRt2/detection_loss‚4.831677405784528 % _timestamp‚1.7885247834313161e+09 _step‚11 1loss/FBt3/assignment_loss‚0.28470510707848007 %loss/total_loss‚95.16224431672711 _runtime‚ 166.966626213 /loss/FRt2/assignment_loss‚7.253244936466217 1loss/FBt1/assignment_loss‚0.41318542598145863 .loss/FRt1/detection_loss‚3.979928587411725 0loss/SRqqt1/detection_loss‚3.059220341620622 /loss/FRt3/assignment_loss‚7.396273740700313 -loss/FRt4/assignment_loss‚7.1564260232644 /loss/FRt4/detection_loss‚2.3614877911864736 /loss/FBt2/detection_loss‚13.929132195810476 2loss/SRqqt3/assignment_loss‚3.9563753694858192 .loss/FRt3/detection_loss‚3.011303629193987 /loss/FBt1/detection_loss‚5.5780050783388075 epoch‚0 0loss/FBt4/assignment_loss‚0.7207480812212452 1loss/SRqqt3/detection_loss‚1.3051273003161281 .loss/FBt3/detection_loss‚5.024452884681523 /loss/FBt4/detection_loss‚1.1910394125793256 trainer/global_step‚149užUéÏ % _timestamp‚1.7885247834313161e+09 _step‚11 1loss/FBt3/assignment_loss‚0.28470510707848007 %loss/total_loss‚95.16224431672711 _runtime‚ 166.966626213 /loss/FRt2/assignment_loss‚7.253244936466217 1loss/FBt1/assignment_loss‚0.41318542598145863 .loss/FRt1/detection_loss‚3.979928587411725 0loss/SRqqt1/detection_loss‚3.059220341620622 /loss/FRt3/assignment_loss‚7.396273740700313 -loss/FRt4/assignment_loss‚7.1564260232644 /loss/FRt4/detection_loss‚2.3614877911864736 /loss/FBt2/detection_loss‚13.929132195810476 2loss/SRqqt3/assignment_loss‚3.9563753694858192 .loss/FRt3/detection_loss‚3.011303629193987 /loss/FBt1/detection_loss‚5.5780050783388075 epoch‚0 0loss/FBt4/assignment_loss‚0.7207480812212452 1loss/SRqqt3/detection_loss‚1.3051273003161281 .loss/FBt3/detection_loss‚5.024452884681523 /loss/FBt4/detection_loss‚1.1910394125793256 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.303622334502464 1loss/SRqqt1/assignment_loss‚3.731844698941266 2loss/SRqqt4/assignment_loss‚3.3092575719801047 0loss/SRqqt2/detection_loss‚2.619613775762461 1loss/SRqqt4/detection_loss‚2.1295591809992063 2loss/SRqqt2/assignment_loss‚3.9653797392117776 0loss/FBt2/assignment_loss‚0.6506337042083032 .loss/FRt2/detection_loss‚4.831677405784528 HNÇ[“ê‚(:ˆ üñêÔœÂËà gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºá Óÿë‚(:ô üñêÔßáÅß network.recv‚152946 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.sdb2.in‚0 cpu‚24.899642053622056 disk.md0.in‚0 disk.md0.out‚ 19.6640625) proc.memory.availableMB‚ 1412.20703125 disk.sda.out‚ 20.4208984375) disk./.usagePercent‚75.87002932055363% disk./.usageGB‚2713.4797439575195 disk.sda.in‚0$ memory_percent‚91.38057231903076 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sdb.out‚ 20.4208984375 disk.sdb2.out‚ 20.4208984375" disk.nvme1n1.out‚ 2750.25390625 disk.sdb.in‚0 disk.sdb1.out‚0 proc.memory.rssMB‚2879.5 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.dm-0.out‚ 2852.30078125 network.sent‚227637 disk.nvme0n1.out‚ 102.046875 disk.sda2.out‚ 20.4208984375 disk.md0p1.in‚0 disk.md0p1.out‚ 19.6640625( proc.memory.percent‚17.5750732421875hW ‘ì‚(:† ‹òêÔû¾à gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üvÆ…÷í‚(:ì ‹òêÔÌÏ®á disk.sda.out‚ 22.287109375 disk.nvme0n1.in‚0) disk./.usagePercent‚75.87351349349592 disk.sda1.in‚0 disk.rbd2.out‚ 0.279296875 disk.sdb2.out‚ 22.287109375 disk.rbd0.in‚0 disk.md0.out‚21.46875 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 2791.88671875 disk.nvme0n1.out‚ 249.828125 disk.rbd0.out‚ 0.279296875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚ 22.287109375 disk.dm-0.out‚ 3020.71484375 network.sent‚278638 network.recv‚158868% disk./.usageGB‚2713.6043548583984 disk.sda2.in‚0 disk.md0.in‚0 disk.sdb.out‚ 22.287109375 disk.sdb2.in‚0 disk.sda.in‚0 disk.dm-0.in‚0$ memory_percent‚91.38057231903076) proc.memory.availableMB‚ 1412.20703125 disk.sdb.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚21.46875( proc.memory.percent‚17.5750732421875 cpu‚24.907227387251954 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 proc.memory.rssMB‚2879.5oI”î‚(:‰ šòêÔºÕ´à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ümvÄ ï‚(:€ šòêÔŒ¶¥á disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚0) proc.memory.percent‚17.57509708404541$ disk./.usageGB‚2713.367160797119 disk.md0p1.in‚0 disk.sda1.in‚0 disk.md0.in‚0$ memory_percent‚91.38486385345459# proc.memory.rssMB‚ 2879.50390625 network.recv‚162415 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚ 0.279296875 cpu‚24.913191326171972) disk./.usagePercent‚75.86688144826913 disk.nvme0n1.out‚ 330.90625 disk.sda2.out‚23.84033203125 proc.cpu.threads‚17 disk.sda.in‚0 disk.md0p1.out‚ 22.94921875 disk.sdb.in‚0) proc.memory.availableMB‚ 1411.50390625 disk.sdb1.in‚0 disk.sda2.in‚0 disk.md0.out‚ 22.94921875 disk.dm-0.out‚ 3291.35546875 disk.sdb2.out‚23.84033203125 disk.sda.out‚23.84033203125 disk.nvme1n1.in‚0 disk.nvme0n1.in‚0 disk.sdb2.in‚0 disk.sdb.out‚23.84033203125 network.sent‚281753" disk.nvme1n1.out‚ 2960.44921875 disk.sdb1.out‚0w‘Ô40ð+ _wandbruntime‚224  _runtime‚224™)È9ÆñÀ _step‚12 ' lr-AdamW/pg1‚0.00012045334616327604 ' lr-AdamW/pg2‚0.00012045334616327604 trainer/global_step‚199 % _timestamp‚1.7885248407759445e+09 _runtime‚ 224.311592301²ÊòÄ ' lr-AdamW/pg1‚0.00012045334616327604 ' lr-AdamW/pg2‚0.00012045334616327604 trainer/global_step‚199 % _timestamp‚1.7885248407759445e+09 _runtime‚ 224.311592301 _step‚12 ØŒ_0ó+ _wandbruntime‚224  _runtime‚224?áùïnôºO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18066155267584 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 224049879584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4042794 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 223141912128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3577147 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1157097 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16347673967104 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4994675 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4994675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4042792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16572566346752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4969829 trainer/global_step‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16347655964160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18066131327488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4042794 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237937664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788524840776859e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.curå  R rent‚ 909204480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 234490101248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18300645368832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 224892379648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2420048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚13 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16049420443568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569¯UŽîÄ'õ¾O WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18066155267584 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 224049879584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4042794 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 223141912128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3577147 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1157097 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16347673967104 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4994675 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4994675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4042792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16572566346752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4969829 trainer/global_step‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16347655964160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18066131327488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4042794 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237937664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788524840776859e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 234490101248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18300645368832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 224892379648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2420048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚13 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16049420443568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 -‹™“ö‚(:ˆ ©òêÔÚ™á gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%y„zý÷‚(:ò ©òêÔ¨…Õá disk.sdb2.in‚0 disk.md0p1.out‚24.6875 cpu‚24.918343048832586) disk./.usagePercent‚75.86685947619951 disk.sdb1.in‚0 disk.sda2.in‚0 disk.md0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚ 3626.9453125) proc.memory.percent‚17.57509708404541 network.recv‚167821 disk.rbd0.out‚ 0.279296875 disk.md0.out‚24.6875 disk.sdb.out‚ 25.6396484375 disk.sda.in‚0# memory_percent‚91.3834810256958) proc.memory.availableMB‚ 1411.73046875 disk.nvme0n1.out‚ 456.6015625 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 3170.34375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sda.out‚ 25.6396484375# proc.memory.rssMB‚ 2879.50390625 proc.cpu.threads‚17 network.sent‚301366 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚0% disk./.usageGB‚2713.3663749694824 disk.sdb2.out‚ 25.6396484375 disk.md0p1.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 25.6396484375vi0ø+ _wandbruntime‚225  _runtime‚225½J¹¶Ë&ùÅM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4062952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 225104635392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16429293052928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4994660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2432115 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16429275049984 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4062954 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 226013853184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18156380972032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15904371119168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885248418739705e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5019532 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5019532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚199 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16655306906112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16128630304574 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 225167153070 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16429275049984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598IíS. úÉM B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9057612 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15904388158528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1162201 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 225104635392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4062954 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 226013853184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18156380972032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15904371119168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885248418739705e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5019532 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5019532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚199 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16655306906112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16128630304574 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 225167153070 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16429275049984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4062952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 225104635392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16429293052928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4994660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2432115 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16429275049984 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20709608062976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20428628258304 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6239577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22577416549888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22577440489984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 $ _timestamp‚1.788524899659647e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050794 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20428610255360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 280979804672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1447066 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5050792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20056177425876 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6239577 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4471011 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 293037987328 _step‚17 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4470364 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569ÛE„ÄÈ'‰ÂO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11265523 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6214731 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 280070600192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22870224988160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20709608062976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20428628258304 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6239577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22577416549888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22577440489984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 $ _timestamp‚1.788524899659647e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050794 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20428610255360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 280979804672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1447066 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5050792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20056177425876 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6239577 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4471011 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 293037987328 _step‚17 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4470364 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0[\Ç.0Š+ _wandbruntime‚284  _runtime‚2849²ä™ ‹ÉM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6264434 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6264434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 _runtime‚ 284.324803173 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6239562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22667642254336 trainer/global_step‚249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 _step‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20510229341184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5070952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22667666194432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 280136440514 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885249007885675e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20510247344128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630*“ë¬Ó&ŒÍM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20792348622336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 282101278208 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 281044408178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22667666194432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 280136440514 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885249007885675e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20510247344128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6264434 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6264434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 _runtime‚ 284.324803173 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6239562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22667642254336 trainer/global_step‚249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 _step‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20510229341184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5070952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 351585356800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24062934408184 % _timestamp‚1.7885249571784549e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23727112979024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27440311069184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/¹£®j inactive_split_bytes.small_pool.freed‚ 351355807744 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5365472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7484479 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27088725712384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 351355807744 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5365472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7484479 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27088725712384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 351585356800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24062934408184 % _timestamp‚1.7885249571784549e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23727112979024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27440311069184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6078954 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13563416 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 % _timestamp‚1.7885249582341647e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27531453230080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27178927476736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24928463117824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 337279485440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7509336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 _runtime‚ 341.771748137 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7509336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24591201635328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3638613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24591183632384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24143069276214 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24591201635328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24591183632384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3638615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚299 _step‚22 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 352751094784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23806130573568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 336921663286 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6078954 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13563416 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 % _timestamp‚1.7885249582341647e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27531453230080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27178927476736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24928463117824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 337279485440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7509336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 _runtime‚ 341.771748137 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7509336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24591201635328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3638613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24591183632384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24143069276214 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24591201635328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24591183632384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3638615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚299 _step‚22 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 352751094784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23806130573568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_(P9(pool.peak‚ 1189001216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 336921663286 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944¦V…0¡+ _wandbruntime‚341  _runtime‚341rà„T¢Î 0loss/FRt2/assignment_loss‚7.2690031711871805 /loss/FBt2/detection_loss‚11.642350886876766 2loss/SRqqt3/assignment_loss‚3.5778999648877043 /loss/FRt2/detection_loss‚3.8551666942926555 /loss/FRt3/detection_loss‚2.8769065402448177 1loss/SRqqt3/detection_loss‚0.9655687397418181 /loss/FBt4/detection_loss‚7.4721613528753785 epoch‚0 # _timestamp‚1.78852495823841e+09 _runtime‚ 341.77386521 0loss/FBt2/assignment_loss‚0.6148940543017278 .loss/FRt1/detection_loss‚3.435236342575239 .loss/FRt4/detection_loss‚2.686229259327606 _step‚23 1loss/FBt1/assignment_loss‚0.47676320298789787 0loss/FBt4/assignment_loss‚0.6203167656293282 %loss/total_loss‚101.9541779423459 /loss/FRt3/assignment_loss‚7.311083810670035 1loss/FBt3/assignment_loss‚0.19081729074597306 1loss/SRqqt2/detection_loss‚3.6304775781252165 2loss/SRqqt1/assignment_loss‚3.1075028562545777 2loss/SRqqt2/assignment_loss‚3.1625770411708136 /loss/FRt1/assignment_loss‚7.355353427969891 1loss/SRqqt4/detection_loss‚2.6101287121574086 /loss/FRt4/assignment_loss‚7.170698245366414 1loss/SRqqt4/assignment_loss‚3.394917933146159 0loss/SRqqt1/detection_loss‚3.111132320761681 /loss/FBt1/detection_loss‚12.946857346365086 /loss/FBt3/detection_loss‚2.4701344046845204 trainer/global_step‚299{íÉLX£Ò 2loss/SRqqt1/assignment_loss‚3.1075028562545777 2loss/SRqqt2/assignment_loss‚3.1625770411708136 /loss/FRt1/assignment_loss‚7.355353427969891 1loss/SRqqt4/detection_loss‚2.6101287121574086 /loss/FRt4/assignment_loss‚7.170698245366414 1loss/SRqqt4/assignment_loss‚3.394917933146159 0loss/SRqqt1/detection_loss‚3.111132320761681 /loss/FBt1/detection_loss‚12.946857346365086 /loss/FBt3/detection_loss‚2.4701344046845204 trainer/global_step‚299 0loss/FRt2/assignment_loss‚7.2690031711871805 /loss/FBt2/detection_loss‚11.642350886876766 2loss/SRqqt3/assignment_loss‚3.5778999648877043 /loss/FRt2/detection_loss‚3.8551666942926555 /loss/FRt3/detection_loss‚2.8769065402448177 1loss/SRqqt3/detection_loss‚0.9655687397418181 /loss/FBt4/detection_loss‚7.4721613528753785 epoch‚0 # _timestamp‚1.78852495823841e+09 _runtime‚ 341.77386521 0loss/FBt2/assignment_loss‚0.6148940543017278 .loss/FRt1/detection_loss‚3.435236342575239 .loss/FRt4/detection_loss‚2.686229259327606 _step‚23 1loss/FBt1/assignment_loss‚0.47676320298789787 0loss/FBt4/assignment_loss‚0.6203167656293282 %loss/total_loss‚101.9541779423459 /loss/FRt3/assignment_loss‚7.311083810670035 1loss/FBt3/assignment_loss‚0.19081729074597306 1loss/SRqqt2/detection_loss‚3.6304775781252165çãϤ‚(:’ ¡óêÔ‰•Ñß gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì:å{ ¥‚(:þ ¡óêÔȶÎâ network.sent‚445999 disk.rbd0.in‚0 disk.md0.out‚ 37.78515625) proc.memory.availableMB‚ 1412.14453125! proc.memory.rssMB‚ 2879.515625 disk.nvme1n1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 cpu‚24.94951902691605 disk.nvme0n1.in‚0 disk.sda2.out‚39.26318359375 disk.dm-0.out‚ 5503.62890625 proc.cpu.threads‚17 network.recv‚208782% disk./.usageGB‚2713.3663635253906 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sdb.out‚39.26318359375 disk.sdb1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚0 disk.sdb.in‚0 disk.sdb2.in‚0 disk.sda.out‚39.26318359375$ memory_percent‚91.38095378875732) disk./.usagePercent‚75.86685915621791 disk.sdb1.out‚0 disk.rbd2.out‚ 0.279296875 disk.sda.in‚0 disk.md0p1.out‚ 37.78515625" disk.nvme0n1.out‚ 1073.31640625 disk.md0.in‚0 disk.sdb2.out‚39.26318359375 disk.nvme1n1.out‚ 4430.3125 disk.md0p1.in‚0) proc.memory.percent‚17.57516860961914á4¿}¦‚(:õ °óêÔó½¢à disk.nvme0n1.out‚ 1116.8125 disk.sda2.in‚0 disk.sda2.out‚40.90869140625 disk.sdb.in‚0 disk.sda.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1411.9921875( disk./.usagePercent‚75.8521120577257 disk.sda.out‚40.90869140625 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0.out‚ 39.3828125 disk.sdb2.in‚0 disk.sdb2.out‚40.90869140625 disk.md0p1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2879.5234375$ disk./.usageGB‚2712.838935852051 disk.rbd2.in‚0 disk.dm-0.out‚ 5652.15625 disk.nvme1n1.out‚ 4535.34375 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.rbd0.in‚0 network.sent‚495664 network.recv‚214774 disk.nvme0n1.in‚0 disk.sdb.out‚40.90869140625 disk.md0p1.out‚ 39.3828125$ memory_percent‚91.38188362121582) proc.memory.percent‚17.57521629333496 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.nvme1n1.in‚0 cpu‚24.953754439381818…pî “§‚(:ˆ °óêÔÚàà gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÂo5“¨‚(:ˆ ¿óêÔ³ÓõÞ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¾Îü©‚(:ñ ¿óêÔÞïáß! disk.nvme0n1.out‚ 1117.4140625 disk.sda2.out‚ 43.3046875 disk.sdb.in‚0 disk.sda.in‚0 disk.md0.out‚ 41.69140625 disk.rbd0.out‚ 0.279296875# memory_percent‚91.3830041885376% disk./.usageGB‚2712.7942657470703 disk.sda1.in‚0 disk.md0.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0 network.sent‚501335) disk./.usagePercent‚75.85086306289456 disk.rbd2.in‚0 disk.dm-0.out‚ 5841.19140625 disk.sdb2.in‚0 disk.sda.out‚ 43.3046875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.sdb2.out‚ 43.3046875" disk.nvme1n1.out‚ 4723.77734375) proc.memory.availableMB‚ 1411.80859375 network.recv‚219096 disk.dm-0.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 41.69140625) proc.memory.percent‚17.57521629333496 disk.nvme0n1.in‚0 disk.sdb.out‚ 43.3046875" proc.memory.rssMB‚ 2879.5234375 cpu‚24.95618487719705 disk.sda2.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17lJ9”ª‚(:‰ ÎóêÔÂÙà gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š±añ«‚(:÷ ÎóêÔØ«¿à disk.dm-0.out‚ 5934.7421875" disk.nvme1n1.out‚ 4745.57421875 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0) disk./.usagePercent‚75.83700828641408 disk.sda.in‚0 disk.md0p1.in‚0 disk.md0.in‚0 network.sent‚507261 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875" disk.nvme0n1.out‚ 1189.16796875 disk.sdb.in‚0) proc.memory.availableMB‚ 1412.02734375) proc.memory.percent‚17.57521629333496 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 44.88671875% disk./.usageGB‚2712.2987518310547 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.279296875 disk.md0.out‚ 43.19921875 cpu‚24.958888376978067 network.recv‚223051 disk.sda.out‚ 44.88671875 disk.md0p1.out‚ 43.19921875 disk.sda2.in‚0" proc.memory.rssMB‚ 2879.5234375 disk.sdb2.in‚0 disk.sdb2.out‚ 44.88671875$ memory_percent‚91.38166904449463 disk.sda2.out‚ 44.88671875ÀÐ"0¬+ _wandbruntime‚394  _runtime‚394îÅ­¿ _runtime‚ 394.050434465 _step‚24 ' lr-AdamW/pg1‚0.00021124732568333338 ' lr-AdamW/pg2‚0.00021124732568333338 trainer/global_step‚349 $ _timestamp‚1.788525010514897e+09Åŵwɮà _step‚24 ' lr-AdamW/pg1‚0.00021124732568333338 ' lr-AdamW/pg2‚0.00021124732568333338 trainer/global_step‚349 $ _timestamp‚1.788525010514897e+09 _runtime‚ 394.050434465oÍ™0¯+ _wandbruntime‚394  _runtime‚394­ëS Ã'°½O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28069691390492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 $ _timestamp‚1.788525010515952e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8704535 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 393154654720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 392245450240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068766383676 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 393154654720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28983691495424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 _runtime‚ 394.05157925 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28983691495424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32010024457728 _step‚25 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 391681644908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4229800 trainer/global_step‚349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8729381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2029573 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31599986994688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28590518837760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15796175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27677992706224 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390773677452 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8729381 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31600010934784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4229798 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28590518837760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 392245450240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068766383676 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 393154654720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28983691495424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 _runtime‚ 394.05157925 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28983691495424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32010024457728 _step‚25 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 391681644908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4229800 trainer/global_step‚349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8729381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2029573 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31599986994688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28590518837760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15796175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27677992706224 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390773677452 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8729381 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31600010934784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4229798 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28590518837760 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28069691390492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 $ _timestamp‚1.788525010515952e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144®£ò0²+ _wandbruntime‚395  _runtime‚395|ߟ j ³ÈM MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7086952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4241865 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27757010300768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 _step‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7086954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28149826258522 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29065504834048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on<€[d_train_batch_end/inactive_split.all.peak‚1221 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8729366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392798918394 _runtime‚ 395.023476861 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31690212699136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27757027340128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29066432054784 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 394276128256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32101664417792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8754238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31690236639232 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 trainer/global_step‚349 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788525011487295e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8754238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2036102 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2035485 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0PŸtÒ&´ÌM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392798918394 _runtime‚ 395.023476861 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31690212699136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27757027340128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29066432054784 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 394276128256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32101664417792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8754238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31690236639232 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 trainer/global_step‚349 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788525011487295e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8754238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2036102 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2035485 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7086952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4241865 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27757010300768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 _step‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7086954 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28149826258522 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29065504834048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8729366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ô•70µ+ _wandbruntime‚395  _runtime‚395ªÝø1Q¶Ë /loss/FRt3/detection_loss‚3.4566570986872134 /loss/FBt4/detection_loss‚1.4693423542468722 2loss/SRqqt2/assignment_loss‚2.4964911469391415 0loss/FBt4/assignment_loss‚0.8017728749443503 /loss/FRt4/detection_loss‚3.0910229238991938 _runtime‚ 395.025095399 2loss/SRqqt1/assignment_loss‚2.2454226409484233 /loss/FBt1/assignment_loss‚0.555936719160527 /loss/FRt2/detection_loss‚3.5964588267462596 0loss/SRqqt1/detection_loss‚2.738863831652062 0loss/SRqqt4/detection_loss‚2.161647675351964 epoch‚0 1loss/FBt3/assignment_loss‚0.35476020416733395 .loss/FBt2/detection_loss‚6.032770190387964 trainer/global_step‚349 1loss/SRqqt3/assignment_loss‚3.414748251438141 0loss/SRqqt2/detection_loss‚2.791156068444252 /loss/FBt1/detection_loss‚6.0239581233263015 /loss/FRt2/assignment_loss‚7.068146773747036 0loss/FRt4/assignment_loss‚7.3087093730767565 0loss/FBt2/assignment_loss‚0.3839979199398537 0loss/SRqqt3/detection_loss‚5.700109723649199 _step‚27 /loss/FRt3/assignment_loss‚7.406637989956399 /loss/FRt1/detection_loss‚2.6109036416338203 .loss/FBt3/detection_loss‚3.645717243045285 /loss/FRt1/assignment_loss‚7.311760266621907 1loss/SRqqt4/assignment_loss‚3.376868393686083 %loss/total_loss‚86.04386025569633 % _timestamp‚1.7885250114896371e+098¿¿@U·Ï 2loss/SRqqt2/assignment_loss‚2.4964911469391415 0loss/FBt4/assignment_loss‚0.8017728749443503 /loss/FRt4/detection_loss‚3.0910229238991938 _runtime‚ 395.025095399 2loss/SRqqt1/assignment_loss‚2.2454226409484233 /loss/FBt1/assignment_loss‚0.555936719160527 /loss/FRt2/detection_loss‚3.5964588267462596 0loss/SRqqt1/detection_loss‚2.738863831652062 0loss/SRqqt4/detection_loss‚2.161647675351964 epoch‚0 1loss/FBt3/assignment_loss‚0.35476020416733395 .loss/FBt2/detection_loss‚6.032770190387964 trainer/global_step‚349 1loss/SRqqt3/assignment_loss‚3.414748251438141 0loss/SRqqt2/detection_loss‚2.791156068444252 /loss/FBt1/detection_loss‚6.0239581233263015 /loss/FRt2/assignment_loss‚7.068146773747036 0loss/FRt4/assignment_loss‚7.3087093730767565 0loss/FBt2/assignment_loss‚0.3839979199398537 0loss/SRqqt3/detection_loss‚5.700109723649199 _step‚27 /loss/FRt3/assignment_loss‚7.406637989956399 /loss/FRt1/detection_loss‚2.6109036416338203 .loss/FBt3/detection_loss‚3.645717243045285 /loss/FRt1/assignment_loss‚7.311760266621907 1loss/SRqqt4/assignment_loss‚3.376868393686083 %loss/total_loss‚86.04386025569633 % _timestamp‚1.7885250114896371e+09 /loss/FRt3/detection_loss‚3.4566570986872134 /loss/FBt4/detection_loss‚1.4693423542468722¸¿hS“¸‚(:ˆ ÝóêÔ©î×à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( ½¹‚(:ø ÝóêÔëœá network.sent‚513707 disk.md0p1.in‚0 disk.dm-0.out‚ 6051.3984375 cpu‚24.960832740979786 proc.cpu.threads‚17 disk.md0p1.out‚ 45.09765625 disk.rbd2.out‚ 0.279296875 disk.md0.in‚0 disk.sdb2.out‚ 46.8544921875 disk.sda.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2879.5234375% disk./.usageGB‚2712.4258003234863 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sdb.in‚0( proc.memory.availableMB‚ 1412.2265625 disk.nvme1n1.in‚0 disk.nvme0n1.out‚1217.375 disk.sda2.out‚ 46.8544921875 disk.sdb.out‚ 46.8544921875 disk.md0.out‚ 45.09765625 disk.dm-0.in‚0 disk.sdb2.in‚0 disk.sda.out‚ 46.8544921875 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0$ memory_percent‚91.38045310974121 network.recv‚228440) disk./.usagePercent‚75.84056061543639! disk.nvme1n1.out‚ 4834.0234375 disk.rbd0.in‚0) proc.memory.percent‚17.57521629333496=‚_±“º‚(:ˆ ìóêÔöÏðÞ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áj…»‚(:ú ìóêÔ΂®à disk.sdb.out‚ 55.1494140625 disk.sda2.out‚ 55.1494140625$ disk./.usageGB‚2712.460235595703 disk.dm-0.out‚ 6130.08984375 disk.sdb2.out‚ 55.1494140625" disk.nvme1n1.out‚ 4835.24609375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 53.31640625 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.out‚ 1294.84375 disk.sda.in‚0 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2879.53125 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sda1.in‚0 network.sent‚565609 network.recv‚235131 disk.dm-0.in‚0 disk.sdb.in‚0 disk.sda.out‚ 55.1494140625 disk.md0p1.in‚0 disk.md0p1.out‚ 53.31640625 disk.sda2.in‚0) disk./.usagePercent‚75.84152344005986 disk.md0.in‚0$ memory_percent‚91.37747287750244) proc.memory.availableMB‚ 1412.71484375 cpu‚24.963372477738673 disk.rbd2.out‚ 0.279296875) proc.memory.percent‚17.57526397705078 proc.cpu.threads‚17`ÛÈZ¼‚(:ú ûóêÔÞ͈à disk.md0p1.out‚ 54.421875 disk.sda1.in‚0 network.sent‚569191 disk.dm-0.in‚0 disk.sdb.in‚0 disk.sdb.out‚56.33154296875 disk.sda.out‚56.33154296875" disk.nvme1n1.out‚ 4835.73046875 disk.sdb1.out‚0" disk.nvme0n1.out‚ 1496.37109375 disk.sda2.out‚56.33154296875 disk.dm-0.out‚ 6332.1015625 disk.sdb2.out‚56.33154296875 disk.sda1.out‚0) proc.memory.percent‚17.57526397705078 disk.sdb1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.md0.out‚ 54.421875 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2879.53125% disk./.usageGB‚2712.8515586853027 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875$ memory_percent‚91.37716293334961 network.recv‚239405 disk.sda.in‚0 disk.rbd2.out‚ 0.279296875 cpu‚24.965146867779204( disk./.usagePercent‚75.8524649974265 disk.md0.in‚0 disk.md0p1.in‚0' proc.memory.availableMB‚ 1412.765625 disk.nvme0n1.in‚0 disk.sdb2.in‚0öŽ+ßž½‚(:“ ûóêÔ°æºà gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01PP„0¾+ _wandbruntime‚447  _runtime‚447¶´Æ¿À ' lr-AdamW/pg1‚0.00024151198552335249 ' lr-AdamW/pg2‚0.00024151198552335249 trainer/global_step‚399 % _timestamp‚1.7885250639295852e+09 _runtime‚ 447.465172775 _step‚28ã3WÆÊÀÄ ' lr-AdamW/pg1‚0.00024151198552335249 ' lr-AdamW/pg2‚0.00024151198552335249 trainer/global_step‚399 % _timestamp‚1.7885250639295852e+09 _runtime‚ 447.465172775 _step‚28Šnœ[0Á+ _wandbruntime‚447  _runtime‚447Xº‹Ã¾O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8074794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9974283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32671473128960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9949437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36111296157184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _runtime‚ 447.46629178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2323787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚29 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7156195 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449242079744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449242079744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33119806004224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9974283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32671473128960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9949437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36111296157184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _runtime‚ 447.46629178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2323787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚29 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7156195 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449242079744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449242079744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33119806004224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36671367138304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31707890027968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4845113 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36671610128384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469869216768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885250648987365e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449454335488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36201497921536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33202546550272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32753092214784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9999140 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32753110217728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448676173502 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32155658233806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚30 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 450363553280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 470088266752 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9999140 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32753110217728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9974268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094954 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9974268 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33203473771008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2329571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _runtime‚ 448.434730549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32753092214784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32155658233806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚30 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 450363553280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 470088266752 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9999140 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32753110217728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9974268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094954 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9974268 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33203473771008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2329571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _runtime‚ 448.434730549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32753092214784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36671367138304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31707890027968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4845113 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36671610128384 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469869216768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885250648987365e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449454335488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36201497921536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33202546550272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32753092214784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9999140 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32753110217728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448676173502 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632ÊØÛ$0Ç+ _wandbruntime‚448  _runtime‚448Óá¥0Ès·J7OÍ /loss/FRt3/detection_loss‚2.2060340411627473 0loss/SRqqt2/detection_loss‚2.262823281414581 trainer/global_step‚399 1loss/SRqqt1/assignment_loss‚2.727201064427694 2loss/SRqqt3/assignment_loss‚3.3790398120880125 /loss/FBt4/assignment_loss‚0.653890215481321 1loss/SRqqt4/detection_loss‚3.0914726306994758 2loss/SRqqt2/assignment_loss‚1.9846129453543462 0loss/FBt3/assignment_loss‚0.4017784029963326 -loss/FBt3/detection_loss‚2.63413042273496 %loss/total_loss‚81.03237872569528 % _timestamp‚1.7885250649009213e+09 _runtime‚ 448.436375807 /loss/FRt3/assignment_loss‚7.360855970809709 1loss/FBt2/assignment_loss‚0.42881242025556016 /loss/FBt1/detection_loss‚4.5915969669818875 epoch‚0 /loss/FRt2/assignment_loss‚7.201110211285678 .loss/FBt2/detection_loss‚3.933190562895366 /loss/FRt4/detection_loss‚2.5968259271918512 1loss/SRqqt3/detection_loss‚2.2594595903205965 /loss/FBt4/detection_loss‚5.4605899179975195 2loss/SRqqt4/assignment_loss‚3.2067258755366006 0loss/FBt1/assignment_loss‚0.2936602178169778 1loss/SRqqt1/detection_loss‚3.4710010517211187 _step‚31 /loss/FRt1/assignment_loss‚7.408832955867686 /loss/FRt4/assignment_loss‚7.165158378834627 /loss/FRt1/detection_loss‚3.2188782292477627 /loss/FRt2/detection_loss‚3.0946976325728675 [WÉÑ -loss/FBt3/detection_loss‚2.63413042273496 %loss/total_loss‚81.03237872569528 % _timestamp‚1.7885250649009213e+09 _runtime‚ 448.436375807 /loss/FRt3/assignment_loss‚7.360855970809709 1loss/FBt2/assignment_loss‚0.42881242025556016 /loss/FBt1/detection_loss‚4.5915969669818875 epoch‚0 /loss/FRt2/assignment_loss‚7.201110211285678 .loss/FBt2/detection_loss‚3.933190562895366 /loss/FRt4/detection_loss‚2.5968259271918512 1loss/SRqqt3/detection_loss‚2.2594595903205965 /loss/FBt4/detection_loss‚5.4605899179975195 2loss/SRqqt4/assignment_loss‚3.2067258755366006 0loss/FBt1/assignment_loss‚0.2936602178169778 1loss/SRqqt1/detection_loss‚3.4710010517211187 _step‚31 /loss/FRt1/assignment_loss‚7.408832955867686 /loss/FRt4/assignment_loss‚7.165158378834627 /loss/FRt1/detection_loss‚3.2188782292477627 /loss/FRt2/detection_loss‚3.0946976325728675 /loss/FRt3/detection_loss‚2.2060340411627473 0loss/SRqqt2/detection_loss‚2.262823281414581 trainer/global_step‚399 1loss/SRqqt1/assignment_loss‚2.727201064427694 2loss/SRqqt3/assignment_loss‚3.3790398120880125 /loss/FBt4/assignment_loss‚0.653890215481321 1loss/SRqqt4/detection_loss‚3.0914726306994758 2loss/SRqqt2/assignment_loss‚1.9846129453543462 0loss/FBt3/assignment_loss‚0.4017784029963326ƒ oF“Ê‚(:ˆ ŠôêÔ¿ÑúÞ gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ­øË‚(:í ŠôêÔ×ÜÝß network.sent‚575184 proc.memory.rssMB‚ 2879.53125 cpu‚24.96734155675346 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.sdb2.in‚0 disk.sda.out‚ 57.478515625' proc.memory.availableMB‚ 1412.515625% disk./.usageGB‚2712.8507194519043 disk.sda2.out‚ 57.478515625 disk.md0.in‚0 disk.dm-0.out‚ 6559.171875 disk.sdb.in‚0 disk.nvme1n1.in‚0$ memory_percent‚91.37868881225586 disk.rbd0.in‚0 disk.sdb.out‚ 57.478515625 disk.sdb2.out‚ 57.478515625) proc.memory.percent‚17.57526397705078 network.recv‚243378 disk.sda1.in‚0 disk.md0p1.in‚0 disk.nvme1n1.out‚ 4835.921875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.md0p1.out‚ 55.50390625) disk./.usagePercent‚75.85244153210942 disk.sdb1.in‚0 disk.nvme0n1.out‚1723.25 disk.sda2.in‚0 disk.md0.out‚ 55.50390625 disk.dm-0.in‚0 disk.sda.in‚0Īr.”Ì‚(:‰ ™ôêÔ³ÚëÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]LœØ Í‚(:€ ™ôêÔÍß³à disk.sdb1.in‚0 cpu‚24.969295514688326$ disk./.usageGB‚2712.917682647705 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sda.out‚58.30322265625 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚249739 disk.sdb.in‚0 disk.sdb.out‚58.30322265625( disk./.usagePercent‚75.8543138510903 disk.sda2.in‚0! disk.nvme1n1.out‚ 4836.2890625) proc.memory.availableMB‚ 1412.65234375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚ 0.279296875" disk.nvme0n1.out‚ 1815.83203125 disk.md0.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚58.30322265625" memory_percent‚91.377854347229 disk.md0p1.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda2.out‚58.30322265625 disk.md0.out‚ 56.26953125 disk.dm-0.out‚ 6652.12109375 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2879.53515625) proc.memory.percent‚17.57528781890869 network.sent‚624801 disk.md0p1.out‚ 56.26953125îê1“΂(:ˆ ¨ôêÔ’¯–ß gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yû¾ÄÏ‚(:ø ¨ôêÔœùÍà disk.sdb2.in‚0 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 4836.4296875 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 6757.1015625 disk.md0p1.out‚ 57.046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.sda.out‚59.11474609375# proc.memory.rssMB‚ 2879.53515625 cpu‚24.97118092860534% disk./.usageGB‚2712.5782775878906 disk.sdb2.out‚59.11474609375 disk.md0p1.in‚0 disk.sdb1.out‚0# memory_percent‚91.3808822631836 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚0 network.recv‚254020) disk./.usagePercent‚75.84482394356583 disk.md0.out‚ 57.046875 disk.sdb.in‚0& proc.memory.availableMB‚ 1412.15625 disk.nvme0n1.out‚ 1920.671875 disk.sda2.out‚59.11474609375 disk.rbd0.in‚0) proc.memory.percent‚17.57528781890869 network.sent‚631096 disk.sda2.in‚0 disk.sdb.out‚59.11474609375}Z"…“Ђ(:ˆ ·ôêÔå®îÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ê,Ñ‚(:ö ·ôêÔ˜’Ìà disk.rbd2.in‚0 disk.sdb2.out‚ 60.552734375 disk.md0.in‚0 disk.dm-0.in‚0! disk.nvme1n1.out‚ 4836.8828125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875$ memory_percent‚91.37957096099854# proc.memory.rssMB‚ 2879.53515625 network.sent‚636036 disk.sdb.out‚ 60.552734375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1412.37109375 proc.cpu.threads‚17 disk.sda2.out‚ 60.552734375 disk.dm-0.out‚ 7011.4140625 disk.md0p1.out‚ 58.4140625 disk.nvme0n1.out‚ 2174.53125% disk./.usageGB‚2712.8531761169434 disk.sda.in‚0 disk.sda.out‚ 60.552734375 disk.nvme1n1.in‚0 disk.nvme0n1.in‚0 cpu‚24.972978440542875 disk.md0.out‚ 58.4140625 disk.md0p1.in‚0) proc.memory.percent‚17.57528781890869 network.recv‚258408 disk.sda2.in‚0 disk.sdb2.in‚0) disk./.usagePercent‚75.85251022149211 disk.sdb.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sda1.in‚0ñdK;0Ò+ _wandbruntime‚502  _runtime‚502òStKÆÓÀ ' lr-AdamW/pg1‚0.00027177664536337157 ' lr-AdamW/pg2‚0.00027177664536337157 trainer/global_step‚449 % _timestamp‚1.7885251187374082e+09 _runtime‚ 502.272932689 _step‚32_˜¡=ÊÔÄ _step‚32 ' lr-AdamW/pg1‚0.00027177664536337157 ' lr-AdamW/pg2‚0.00027177664536337157 trainer/global_step‚449 % _timestamp‚1.7885251187374082e+09 _runtime‚ 502.272932689 ‹RUþ0Õ+ _wandbruntime‚502  _runtime‚502²)¥ÄÉ'ÖÃO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11194339 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35579769199984 % _timestamp‚1.7885251187381234e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36752427420160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20277131 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11219185 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 505329504768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 504420300288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 _step‚33 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9082792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35579752160624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 505329504768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41150141770240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5436300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 502528187668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36752427420160 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36083205355108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41149896669696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36082280348292 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37256847720448 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11219185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1226 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9082792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 527339230208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 504420300288 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40622581379584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36752445423104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11194339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 503436155124 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8053645 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2617345 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37257774927872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37257774927872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11219185 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 505329504768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 504420300288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 _step‚33 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9082792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35579752160624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 505329504768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41150141770240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5436300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 502528187668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36752427420160 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36083205355108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41149896669696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36082280348292 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456!_•v´0Ø+ _wandbruntime‚503  _runtime‚503P%Ò ÙÌM WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 505541760512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1226 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41241537120256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35658786794528 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 503645460946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 trainer/global_step‚449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9102952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259767296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 528494209024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5448363 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2622519 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235827200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788525119748756e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 _runtime‚ 503.284809662 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/segment.small_poolû_.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 506450978304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11244042 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 _step‚34 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20322122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36163340223138 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36162415216114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 504553428610 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20346996 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11219170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2623138 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1189 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20322122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36163340223138 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36162415216114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 504553428610 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20346996 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11219170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2623138 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1189 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 _runtime‚ 503.284809662 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 506450978304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11244042 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 _step‚34"ðˆý0Û+ _wandbruntime‚503  _runtime‚503k’öTÜÎ .loss/FRt1/detection_loss‚3.461935981363058 /loss/FBt3/detection_loss‚3.3568453816015547 epoch‚0 0loss/FRt2/assignment_loss‚7.0583930015563965 .loss/FRt3/detection_loss‚2.586480788620455 /loss/FRt4/detection_loss‚2.8974587956861573 1loss/SRqqt1/detection_loss‚2.7993028950691223 1loss/SRqqt3/detection_loss‚2.4035254289527406 -loss/FBt1/detection_loss‚5.19282050182422 1loss/FBt1/assignment_loss‚0.45465149825637735 1loss/FBt3/assignment_loss‚0.48924819340009423 %loss/total_loss‚82.84875818520908 /loss/FRt3/assignment_loss‚7.408839430127825 2loss/SRqqt4/assignment_loss‚3.1809379450032407 0loss/SRqqt2/detection_loss‚3.831078357421435 trainer/global_step‚449 _runtime‚ 503.286795501 /loss/FRt4/assignment_loss‚7.224832787805674 1loss/SRqqt4/detection_loss‚2.0634772802742436 _step‚35 1loss/SRqqt3/assignment_loss‚3.555491044407799 0loss/FBt4/assignment_loss‚0.7106274008750916 /loss/FRt2/detection_loss‚3.6785035783594306 .loss/FBt4/detection_loss‚4.868992838030681 2loss/SRqqt2/assignment_loss‚1.9308360249568255 1loss/FBt2/assignment_loss‚0.20071065908094474 % _timestamp‚1.7885251197515128e+09 /loss/FRt1/assignment_loss‚7.351922619342804 .loss/FBt2/detection_loss‚3.770775556564331 2loss/SRqqt1/assignment_loss‚2.3710701966285708.´{ÎXÝÒ 1loss/SRqqt3/assignment_loss‚3.555491044407799 0loss/FBt4/assignment_loss‚0.7106274008750916 /loss/FRt2/detection_loss‚3.6785035783594306 .loss/FBt4/detection_loss‚4.868992838030681 2loss/SRqqt2/assignment_loss‚1.9308360249568255 1loss/FBt2/assignment_loss‚0.20071065908094474 % _timestamp‚1.7885251197515128e+09 /loss/FRt1/assignment_loss‚7.351922619342804 .loss/FBt2/detection_loss‚3.770775556564331 2loss/SRqqt1/assignment_loss‚2.3710701966285708 .loss/FRt1/detection_loss‚3.461935981363058 /loss/FBt3/detection_loss‚3.3568453816015547 epoch‚0 0loss/FRt2/assignment_loss‚7.0583930015563965 .loss/FRt3/detection_loss‚2.586480788620455 /loss/FRt4/detection_loss‚2.8974587956861573 1loss/SRqqt1/detection_loss‚2.7993028950691223 1loss/SRqqt3/detection_loss‚2.4035254289527406 -loss/FBt1/detection_loss‚5.19282050182422 1loss/FBt1/assignment_loss‚0.45465149825637735 1loss/FBt3/assignment_loss‚0.48924819340009423 %loss/total_loss‚82.84875818520908 /loss/FRt3/assignment_loss‚7.408839430127825 2loss/SRqqt4/assignment_loss‚3.1809379450032407 0loss/SRqqt2/detection_loss‚3.831078357421435 trainer/global_step‚449 _runtime‚ 503.286795501 /loss/FRt4/assignment_loss‚7.224832787805674 1loss/SRqqt4/detection_loss‚2.0634772802742436 _step‚35#^Zd“Þ‚(:ˆ ÆôêÔŽÝ½ß gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ç¡ß‚(:õ ÆôêÔ”“±à disk.dm-0.out‚ 7188.17578125" disk.nvme1n1.out‚ 4837.07421875 disk.nvme0n1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.sdb2.in‚0 disk.nvme1n1.in‚0$ memory_percent‚91.38147830963135 disk.sda1.in‚0 disk.sdb.in‚0 disk.sdb2.out‚ 61.541015625 disk.sda.out‚ 61.541015625 disk.rbd0.out‚ 0.279296875) proc.memory.percent‚17.57528781890869 proc.cpu.threads‚17 disk.md0.out‚59.34375 disk.sdb.out‚ 61.541015625 disk.sda.in‚0) proc.memory.availableMB‚ 1412.05859375 network.sent‚642133) disk./.usagePercent‚75.86658823846632$ disk./.usageGB‚2713.356674194336! disk.nvme0n1.out‚ 2351.1015625 disk.sda2.out‚ 61.541015625 disk.md0.in‚0 disk.dm-0.in‚0 disk.sdb1.in‚0 network.recv‚262680 disk.rbd2.in‚0 disk.sda2.in‚0 disk.md0p1.out‚59.34375# proc.memory.rssMB‚ 2879.53515625 disk.md0p1.in‚0 disk.rbd0.in‚0 cpu‚24.974694182256066yž@÷’à‚(:‡ ÕôêÔ„„Åà gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú‰b5á‚(:û ÕôêÔóê©á disk.md0p1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2879.5390625 disk.sdb.in‚0 disk.sdb.out‚62.43896484375 network.recv‚269241 disk.sda1.out‚0) proc.memory.availableMB‚ 1412.28515625 proc.cpu.threads‚17) disk./.usagePercent‚75.87353333235491# disk./.usageGB‚2713.60506439209 disk.sda.out‚62.43896484375 disk.sdb1.out‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 2576.55859375 disk.sda2.out‚62.43896484375 disk.md0.in‚0 disk.dm-0.in‚0! disk.nvme1n1.out‚ 4837.2734375 disk.md0.out‚ 60.171875$ memory_percent‚91.38009548187256 cpu‚24.97620611845035 disk.nvme1n1.in‚0 disk.md0p1.out‚ 60.171875 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 7413.83203125 disk.sdb2.in‚0 disk.sdb2.out‚62.43896484375( proc.memory.percent‚17.5753116607666 network.sent‚693808 disk.sda.in‚0ÍWGâ‚(:’ äôêÔ£ç½à gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬& ã‚(:€ äôêԪݗá) disk./.usagePercent‚75.87351893318306 disk.sdb.out‚63.59521484375 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sdb2.out‚63.59521484375 disk.md0p1.out‚ 61.2578125 disk.rbd0.in‚0$ memory_percent‚91.37997627258301" disk.nvme0n1.out‚ 2841.34765625 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 1412.3046875 disk.sdb.in‚0$ disk./.usageGB‚2713.604549407959 disk.sda1.in‚0 disk.rbd2.out‚ 0.279296875 disk.md0.in‚0 disk.dm-0.out‚ 7678.76171875# proc.memory.rssMB‚ 2879.54296875 disk.sda2.out‚63.59521484375 disk.sdb2.in‚0 disk.sda.out‚63.59521484375 disk.md0p1.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 4837.4140625 cpu‚24.976853064567344 network.sent‚698147 network.recv‚273925 disk.sda1.out‚0 disk.dm-0.in‚0) proc.memory.percent‚17.57533550262451 disk.rbd0.out‚ 0.279296875 disk.md0.out‚ 61.2578125>n&žä‚(:“ óôêÔ¿¾©à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<÷»å‚(:û óôêÔÈò¢á) proc.memory.availableMB‚ 1412.03515625# proc.memory.rssMB‚ 2879.54296875 disk.sdb2.out‚ 64.6220703125$ disk./.usageGB‚2713.854549407959 disk.dm-0.out‚ 7894.81640625 disk.sdb.out‚ 64.6220703125 disk.nvme1n1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.sdb2.in‚0 disk.md0.out‚ 62.2265625 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 64.6220703125) proc.memory.percent‚17.57533550262451 disk.sdb.in‚0 disk.sda.in‚0 disk.sdb1.out‚0 cpu‚24.97744374259029 network.sent‚706181 network.recv‚278268 disk.md0.in‚0 disk.rbd0.out‚ 0.279296875! disk.nvme1n1.out‚ 4837.7109375 disk.rbd2.out‚ 0.279296875" disk.nvme0n1.out‚ 3057.10546875$ memory_percent‚91.38162136077881 disk.sda2.in‚0( disk./.usagePercent‚75.8805090378162 disk.sda.out‚ 64.6220703125 disk.md0p1.in‚0 disk.md0p1.out‚ 62.2265625 disk.sdb1.in‚0 disk.sda1.in‚0•ésn0æ+ _wandbruntime‚560  _runtime‚560®@_äÄç¾ _step‚36 & lr-AdamW/pg1‚0.0003020413052033907 & lr-AdamW/pg2‚0.0003020413052033907 trainer/global_step‚499 % _timestamp‚1.7885251772551203e+09 _runtime‚ 560.790766935ÌO^ûÈè trainer/global_step‚499 % _timestamp‚1.7885251772551203e+09 _runtime‚ 560.790766935 _step‚36 & lr-AdamW/pg1‚0.0003020413052033907 & lr-AdamW/pg2‚0.0003020413052033907$EˆGV0é+ _wandbruntime‚560  _runtime‚560'g™mêÇO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40833381711360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12439241 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41394816644096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560507725312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41394816644096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41393889436672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/ac”(TÚ` tive.large_pool.peak‚3725 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 559313410232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 561416929792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10090794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45133866601984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40833399714304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 586113393152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885251772563045e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12464087 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 561416929792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 558405442776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8950834 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2910627 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40833381711360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚ 560.792235802:¨­ÀÑ'ëËO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40833381711360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12439241 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41394816644096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560507725312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41394816644096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41393889436672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 559313410232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 561416929792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10090794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45133866601984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40833399714304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 586113393152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885251772563045e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12464087 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 561416929792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 558405442776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8950834 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2910627 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40833381711360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚ 560.792235802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚499 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45224092306432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2917214 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 $ _timestamp‚1.788525178429972e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110954 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6051615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 587517576192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚ 561.967782074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110952 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41477557203456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 562538403328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10110952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚38 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8968829 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 562538403328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40915018800128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40170097205446 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720R™w¦$îÔM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41477557203456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 562538403328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10110952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚38 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8968829 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 562538403328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40915018800128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40170097205446 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12464072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40915000797184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 561629185536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22575024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚499 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45224092306432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2917214 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 $ _timestamp‚1.788525178429972e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110954 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6051615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 587517576192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚ 561.967782074dìå4 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110952 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13708989 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49645127884288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43481511615024 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 615190665340 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13684143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45530931152896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 616595150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098794 _runtime‚ 619.299131028 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44914336002560 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6642798 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9849080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 645327577600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44096719319724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 266072064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚41 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44914354005504åÚTÑ'ÿËO % _timestamp‚1.7885252357630897e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13708989 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49645127884288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43481511615024 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 615190665340 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13684143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45530931152896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 616595150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098794 _runtime‚ 619.299131028 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44914336002560 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6642798 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9849080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 645327577600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44096719319724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 266072064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚41 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44914354005504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 616595150336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24807783 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50290213329920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44176854187754 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 618625828352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50381856181760 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11118952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13733846 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13708974 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 trainer/global_step‚549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44995973091328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 615399971162 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13708974 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50381602705920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118954 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9867711 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 " _timestamp‚1.7885252367623e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49735353588736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44995955088384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13733846 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654865 _runtime‚ 620.298599471 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800:óÙøØ&‚ÒM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13733846 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13708974 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 trainer/global_step‚549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44995973091328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 615399971162 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13708974 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50381602705920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118954 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9867711 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 " _timestamp‚1.7885252367623e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49735353588736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44995955088384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13733846 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654865 _runtime‚ 620.298599471 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49735377528832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44176854187754 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 618625828352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50381856181760 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11118952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845*é–H0ƒ+ _wandbruntime‚620  _runtime‚620Ç=üP„Ê 1loss/SRqqt4/assignment_loss‚2.857524141021397 /loss/FBt3/detection_loss‚1.2801590605582482 epoch‚0 1loss/SRqqt3/assignment_loss‚3.422464680671692 0loss/FBt4/assignment_loss‚0.8637363786498705 0loss/FRt3/assignment_loss‚7.4492037589089914 % _timestamp‚1.7885252367643976e+09 0loss/FRt4/assignment_loss‚6.5812216411937365 1loss/SRqqt1/assignment_loss‚2.329315980275472 -loss/FRt2/detection_loss‚2.67086900472641 1loss/SRqqt3/detection_loss‚1.7171267295854817 .loss/FBt2/detection_loss‚4.717290805144743 .loss/FBt4/detection_loss‚3.923788971371121 2loss/SRqqt2/assignment_loss‚2.6329239404807656 0loss/FBt1/assignment_loss‚0.8312772214412689 0loss/FBt2/assignment_loss‚0.6156116084222792 1loss/FBt3/assignment_loss‚0.30765199001916815 $loss/total_loss‚78.2564155319103 trainer/global_step‚549 _runtime‚ 620.299672365 /loss/FRt1/assignment_loss‚6.741942272629848 /loss/FRt2/assignment_loss‚6.679289031028747 /loss/FRt1/detection_loss‚3.0074840465257333 1loss/SRqqt1/detection_loss‚2.2768258263667422 1loss/SRqqt2/detection_loss‚2.3391413115343807 0loss/SRqqt4/detection_loss‚3.018107745958411 .loss/FRt3/detection_loss‚2.529761574247427 /loss/FRt4/detection_loss‚2.4600100181319497 .loss/FBt1/detection_loss‚7.003687793016434 _step‚43Ø·ø\T…Î 0loss/FRt3/assignment_loss‚7.4492037589089914 % _timestamp‚1.7885252367643976e+09 0loss/FRt4/assignment_loss‚6.5812216411937365 1loss/SRqqt1/assignment_loss‚2.329315980275472 -loss/FRt2/detection_loss‚2.67086900472641 1loss/SRqqt3/detection_loss‚1.7171267295854817 .loss/FBt2/detection_loss‚4.717290805144743 .loss/FBt4/detection_loss‚3.923788971371121 2loss/SRqqt2/assignment_loss‚2.6329239404807656 0loss/FBt1/assignment_loss‚0.8312772214412689 0loss/FBt2/assignment_loss‚0.6156116084222792 1loss/FBt3/assignment_loss‚0.30765199001916815 $loss/total_loss‚78.2564155319103 trainer/global_step‚549 _runtime‚ 620.299672365 /loss/FRt1/assignment_loss‚6.741942272629848 /loss/FRt2/assignment_loss‚6.679289031028747 /loss/FRt1/detection_loss‚3.0074840465257333 1loss/SRqqt1/detection_loss‚2.2768258263667422 1loss/SRqqt2/detection_loss‚2.3391413115343807 0loss/SRqqt4/detection_loss‚3.018107745958411 .loss/FRt3/detection_loss‚2.529761574247427 /loss/FRt4/detection_loss‚2.4600100181319497 .loss/FBt1/detection_loss‚7.003687793016434 _step‚43 1loss/SRqqt4/assignment_loss‚2.857524141021397 /loss/FBt3/detection_loss‚1.2801590605582482 epoch‚0 1loss/SRqqt3/assignment_loss‚3.422464680671692 0loss/FBt4/assignment_loss‚0.8637363786498705+¡”°¿’†‚(:‡ ¾õêÔþ„äà gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô˜[ ‡‚(:þ ¾õêÔýü»á disk.rbd0.out‚ 0.279296875! disk.nvme0n1.out‚ 4200.2734375 disk.dm-0.out‚ 9039.265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.md0p1.out‚ 67.27734375! disk.nvme1n1.out‚ 4838.9921875 disk.md0p1.in‚0# proc.memory.rssMB‚ 2879.54296875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.sda2.out‚69.98388671875( proc.memory.availableMB‚ 1412.1796875) proc.memory.percent‚17.57533550262451 disk.sdb.in‚0 disk.sdb2.in‚0 disk.sda.out‚69.98388671875 disk.sdb2.out‚69.98388671875 network.sent‚777917 network.recv‚302870 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 67.27734375 cpu‚24.981161675168604 disk.nvme1n1.in‚0 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚0$ memory_percent‚91.38073921203613 disk.sdb.out‚69.98388671875( disk./.usagePercent‚75.8700275073246$ disk./.usageGB‚2713.479679107666• d¼”ˆ‚(:‰ ÍõêÔæïà gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â™kŠ‰‚(:„ ÍõêÔ„¥«á disk.sda1.in‚0) proc.memory.percent‚17.57533550262451) disk./.usagePercent‚75.87352266630168 disk.dm-0.in‚0 disk.sdb.in‚0 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875) proc.memory.availableMB‚ 1412.17578125# proc.memory.rssMB‚ 2879.54296875 network.sent‚829418 disk.dm-0.out‚ 9320.73046875 disk.rbd2.in‚0 disk.sda2.in‚0$ memory_percent‚91.38076305389404 network.recv‚309221 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.sda2.out‚71.38818359375 disk.sdb.out‚71.38818359375" disk.nvme1n1.out‚ 4839.26953125 disk.sdb1.out‚0 disk.sda1.out‚0 cpu‚24.982341863770486 disk.sdb1.in‚0 disk.sda.in‚0 disk.sda.out‚71.38818359375 disk.md0.out‚ 68.61328125 disk.sdb2.out‚71.38818359375 disk.md0p1.in‚0! disk.nvme0n1.out‚ 4481.4609375 proc.cpu.threads‚17 disk.nvme1n1.in‚0% disk./.usageGB‚2713.6046829223633 disk.md0p1.out‚ 68.61328125 disk.md0.in‚0 ¿H·”Š‚(:‰ ÜõêÔæòµà gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.793' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.44825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12106792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 704142114816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47432391342224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7246048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48995290293760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54860579161600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7246050 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54156413106688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54156437046784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14929045 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49668900076544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14953891 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 % _timestamp‚1.7885252940827897e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27060685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3501779 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14929045 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48995308296704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48102551295216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0e3»fÓ'“ÍO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14929045 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48995308296704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48102551295216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 672682575360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10747185 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚ 677.618942106 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3501137 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27035837 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54860334061056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47432408381584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12106792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48995290293760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48103476302032 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 670159952992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 671067920448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12106794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12106792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 704142114816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47432391342224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7246048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48995290293760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54860579161600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7246050 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54156413106688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54156437046784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14929045 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49668900076544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14953891 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 % _timestamp‚1.7885252940827897e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27060685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3501779 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570-†Ç*ý0”+ _wandbruntime‚678  _runtime‚678¾ªæ‚Ù&•ÓM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49076927382528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12126954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7258115 _step‚46 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49751640635904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49076909379584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚ 678.800722041 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 705111422976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48182686163038 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10765712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54951750234112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47511408936768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 674713253376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 672185193934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 674713253376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¡ev Y"–×M F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12126954 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49076909379584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12126952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27080828 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54951991127040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚ 678.800722041 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 705111422976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48182686163038 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10765712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54951750234112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47511408936768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 674713253376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 672185193934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 674713253376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49076927382528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12126954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7258115 _step‚46 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49751640635904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49076909379584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F 0¥+ _wandbruntime‚735  _runtime‚7356U5ÞÏ'¦ÉO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53076262587904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11645681 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53076262587904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53076244584960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3797048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 726945175556 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51383288108784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728770000384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16173947 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59430561502720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚49 _runtime‚ 735.927840112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53076244584960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29313587 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16198793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7849300 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885253523922007e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13114794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3796383 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58667698329088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11646348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52110233284340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53805941792768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16173947 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51383271069424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59430814991872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53805014585344 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729679204864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726037208100 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 729679204864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760ËB|—Ó'§ÍO _runtime‚ 735.927840112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53076244584960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29313587 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16198793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7849300 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885253523922007e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13114794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3796383 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58667698329088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11646348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52110233284340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53805941792768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16173947 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51383271069424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59430814991872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53805014585344 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729679204864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726037208100 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 729679204864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53076262587904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11645681 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53076262587904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53076244584960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3797048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 726945175556 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51383288108784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728770000384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16173947 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59430561502720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚491m Bz0¨+ _wandbruntime‚737  _runtime‚737h.4À ©ÓM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16198778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727154481378 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53888682352128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53888682352128 [JDeviceStatsMonitor.on_tr¢eLÌain_batch_end/allocated_bytes.small_pool.allocated‚ 730800678400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 764268059648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885253534748535e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16223650 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7861363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52189443145346 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚50 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚ 737.013208421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11664255 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53157863670784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53887755131392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Q®Ò Ý&ª×M M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53887755131392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16198778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727154481378 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53888682352128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53888682352128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 730800678400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 764268059648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885253534748535e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16223650 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7861363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52189443145346 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚50 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚ 737.013208421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11664255 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53157863670784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 9079676642ÂÉÁ0«+ _wandbruntime‚737  _runtime‚737ƒÃæN¬È 1loss/SRqqt2/detection_loss‚2.3948857600872335 trainer/global_step‚649 % _timestamp‚1.7885253534798515e+09 2loss/SRqqt3/assignment_loss‚3.4255266338586807 epoch‚0 /loss/FRt1/assignment_loss‚7.247106300459968 .loss/FRt3/assignment_loss‚4.48214841740472 0loss/FBt2/assignment_loss‚0.6603485314062709 0loss/FBt4/assignment_loss‚1.4998440146446228 .loss/FRt1/detection_loss‚3.038273570438226 .loss/FBt2/detection_loss‚4.482001194730401 _runtime‚ 737.01537025 2loss/SRqqt2/assignment_loss‚1.7775951490665858 /loss/FRt2/assignment_loss‚6.059471145752938 0loss/FBt1/assignment_loss‚0.8617246896019424 /loss/FRt3/detection_loss‚2.2205750633563315 1loss/SRqqt4/assignment_loss‚2.165937630886055 .loss/FBt4/detection_loss‚6.732788950204849 %loss/total_loss‚74.54233087675188 _step‚51 2loss/SRqqt1/assignment_loss‚3.0501288147859795 1loss/SRqqt4/detection_loss‚1.6794764769364552 .loss/FRt2/detection_loss‚2.858849225505706 /loss/SRqqt3/detection_loss‚3.16375812003389 /loss/FBt1/detection_loss‚4.9452332481741905 /loss/FBt3/detection_loss‚1.5103675998239017 .loss/FRt4/assignment_loss‚5.23830560726278 0loss/FBt3/assignment_loss‚0.3153975230453499 /loss/FRt4/detection_loss‚2.0596049809280563 1loss/SRqqt1/detection_loss‚2.6729822283567386Þ¥‘ìR­Ì epoch‚0 /loss/FRt1/assignment_loss‚7.247106300459968 .loss/FRt3/assignment_loss‚4.48214841740472 0loss/FBt2/assignment_loss‚0.6603485314062709 0loss/FBt4/assignment_loss‚1.4998440146446228 .loss/FRt1/detection_loss‚3.038273570438226 .loss/FBt2/detection_loss‚4.482001194730401 _runtime‚ 737.01537025 2loss/SRqqt2/assignment_loss‚1.7775951490665858 /loss/FRt2/assignment_loss‚6.059471145752938 0loss/FBt1/assignment_loss‚0.8617246896019424 /loss/FRt3/detection_loss‚2.2205750633563315 1loss/SRqqt4/assignment_loss‚2.165937630886055 .loss/FBt4/detection_loss‚6.732788950204849 %loss/total_loss‚74.54233087675188 _step‚51 2loss/SRqqt1/assignment_loss‚3.0501288147859795 1loss/SRqqt4/detection_loss‚1.6794764769364552 .loss/FRt2/detection_loss‚2.858849225505706 /loss/SRqqt3/detection_loss‚3.16375812003389 /loss/FBt1/detection_loss‚4.9452332481741905 /loss/FBt3/detection_loss‚1.5103675998239017 .loss/FRt4/assignment_loss‚5.23830560726278 0loss/FBt3/assignment_loss‚0.3153975230453499 /loss/FRt4/detection_loss‚2.0596049809280563 1loss/SRqqt1/detection_loss‚2.6729822283567386 1loss/SRqqt2/detection_loss‚2.3948857600872335 trainer/global_step‚649 % _timestamp‚1.7885253534798515e+09 2loss/SRqqt3/assignment_loss‚3.42552663385868073xŒÿÌ’®‚(:‡ ¶öêÔÎŽÏà gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¼ò¢ ¯‚(:ÿ ¶öêÔ­¤Ðá# disk./.usageGB‚2713.85595703125 disk.rbd0.out‚ 0.279296875 disk.sda.out‚79.63427734375 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 0.279296875 disk.sda2.out‚79.63427734375 disk.md0.in‚0 disk.sdb.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1411.88671875 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 76.4296875 disk.md0p1.out‚ 76.4296875" disk.nvme0n1.out‚ 5880.36328125 disk.dm-0.in‚0" disk.nvme1n1.out‚ 5147.45703125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 11027.8203125 disk.sdb.out‚79.63427734375# memory_percent‚91.3825273513794" proc.memory.rssMB‚ 2879.5546875* proc.memory.percent‚17.575407028198242 network.sent‚923761 network.recv‚339618 cpu‚24.98891701208825 proc.cpu.threads‚17) disk./.usagePercent‚75.88054839555255 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚79.63427734375 disk.md0p1.in‚0 disk.sda2.in‚0Ž6Ìñ“°‚(:ˆ ÅöêÔÇÏúà gpu.0.gpu‚35.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07øsø±‚(:í ÅöêÔؼ¶á network.sent‚961296 disk.sda.in‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚77.4375 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 5907.34375 disk.sdb2.out‚ 80.69921875 disk.sda.out‚ 80.69921875 disk.rbd0.in‚0 disk.sdb1.in‚0 cpu‚24.989775171213935 disk.sda2.in‚0 disk.sda2.out‚ 80.69921875 disk.dm-0.out‚11173.31640625 disk.sdb2.in‚0 disk.rbd2.out‚ 0.279296875 proc.cpu.threads‚17 disk.sdb.in‚0 disk.sdb.out‚ 80.69921875 disk.md0p1.in‚0 disk.md0.out‚77.4375 disk.dm-0.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1411.8828125* proc.memory.percent‚17.575407028198242) disk./.usagePercent‚75.88051479748493% disk./.usageGB‚2713.8547554016113" disk.nvme1n1.out‚ 5276.97265625# memory_percent‚91.3825511932373" proc.memory.rssMB‚ 2879.5546875 network.recv‚344807 disk.md0.in‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.sda1.out‚0—”£¸“²‚(:ˆ ÔöêÔ–ÏÃà gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃqæÉ ³‚(:€ ÔöêÔ¢ÒÛà disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1412.2109375 disk.sdb.out‚82.03955078125 disk.sdb2.in‚0 disk.md0p1.out‚ 78.71484375 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.279296875 disk.dm-0.in‚0* proc.memory.percent‚17.575454711914062$ disk./.usageGB‚2713.854804992676 disk.sdb.in‚0 disk.nvme0n1.in‚0 proc.memory.rssMB‚ 2879.5625) disk./.usagePercent‚75.88051618407185 disk.sda1.out‚0! disk.nvme0n1.out‚ 6037.8828125 network.recv‚349090 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 5341.16015625 disk.sda2.in‚0 disk.md0.in‚0 disk.sda.in‚0 disk.sda2.out‚82.03955078125 cpu‚24.990012055517056 network.sent‚965239 disk.md0.out‚ 78.71484375 disk.sdb2.out‚82.03955078125 disk.sda.out‚82.03955078125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚11379.04296875$ memory_percent‚91.38054847717285 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.rbd0.in‚0 F}O0´+ _wandbruntime‚794  _runtime‚794¡v…¦Äµ¾ _runtime‚ 794.614466926 _step‚52 & lr-AdamW/pg1‚0.0004230999445634671 & lr-AdamW/pg2‚0.0004230999445634671 trainer/global_step‚699 % _timestamp‚1.7885254110783374e+09ZÛ–Sȶ _runtime‚ 794.614466926 _step‚52 & lr-AdamW/pg1‚0.0004230999445634671 & lr-AdamW/pg2‚0.0004230999445634671 trainer/global_step‚699 % _timestamp‚1.7885254110783374e+094[öô0·+ _wandbruntime‚794  _runtime‚794^¥¡Žf¸ÉO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784857425408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17443695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 794.615782392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63178983551488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14122794 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000692171776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8452548 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚699 % _timestamp‚1.7885254110800347e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 821708620288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782822430664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4092947 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14122792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57942983508992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848yôÓ'¹ÍO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4092947 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14122792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57942983508992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784857425408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17443695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 794.615782392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63178983551488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14122794 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000692171776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8452548 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚699 % _timestamp‚1.7885254110800347e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 821708620288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7828224306645?ï¢]’º‚(:‡ ãöêÔ«žâà gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÏù»‚(:î ãöêÔ Šäá" disk.nvme1n1.out‚ 5384.89453125! disk.nvme0n1.out‚ 6300.0234375 disk.dm-0.out‚11684.91796875 network.recv‚353154 disk.sdb1.out‚0 proc.memory.rssMB‚ 2879.5625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 83.3984375 disk.sda.out‚ 83.3984375 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.md0.out‚ 80.00390625 disk.dm-0.in‚0 cpu‚24.990737108714676 network.sent‚970553 disk.rbd0.in‚0 disk.sda2.out‚ 83.3984375 disk.md0.in‚0 disk.sdb2.in‚0 disk.sda.in‚0 disk.sda1.out‚0$ disk./.usageGB‚2713.327823638916 disk.sdb.in‚0 disk.sdb.out‚ 83.3984375 disk.md0p1.in‚0 disk.md0p1.out‚ 80.00390625 disk.rbd0.out‚ 0.279296875& proc.memory.availableMB‚ 1411.96875* proc.memory.percent‚17.575454711914062 disk.rbd2.out‚ 0.279296875$ memory_percent‚91.38202667236328 proc.cpu.threads‚17) disk./.usagePercent‚75.86578156486189 disk.nvme1n1.in‚0Ä¡²0¼+ _wandbruntime‚795  _runtime‚795Ãò@9Ø&½ÒM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14142954 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786888103424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56197125134678 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56196200127654 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12563522 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58025724068352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 822881187328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 $ _timestamp‚1.788525412093928e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4098242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 785978885632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4098907 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 823106528768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55413168391168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64092090443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55413185430528 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 783939704150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57238835964928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17443680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63269209255936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31586632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57238817961984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57238817961984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 822881187328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 $ _timestamp‚1.788525412093928e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4098242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 785978885632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4098907 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 823106528768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55413168391168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64092090443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55413185430528 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 783939704150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57238835964928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17443680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63269209255936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31586632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57238817961984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57238817961984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 840944850432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67690268773888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33794543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62080025225216 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 838699685772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚57 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15130792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4389320 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 841854054912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15130794 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59285047563184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456®A¿8Ó'ÍÍO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 840944850432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67690268773888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33794543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62080025225216 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 838699685772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚57 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15130792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4389320 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 841854054912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15130794 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59285047563184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61238153167360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 837791718316 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚ 853.251002309 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33819391 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18663751 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚749 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9055800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 840944850432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60123747248956 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 255586304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62079098017792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68571246928896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61238171170304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67690292713984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 231646208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61238171170304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 842066310656 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 881894799872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60203882116986 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59364065157728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pooÒÇU‘l.current‚24872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 _runtime‚ 854.206649282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚58 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 842066310656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62161838563840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4394434 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67780518418432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15150952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18688582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61319790256128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18688582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59364048118368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60202957109962 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4395098 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 882122238464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570`­ÉÜ&ÐÖM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60202957109962 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4395098 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 882122238464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18713454 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61319772253184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15150952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68662389278208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 $ _timestamp‚1.788525470670486e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62162765784576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 842066310656 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 881894799872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60203882116986 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59364065157728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 _runtime‚ 854.206649282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚58 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 842066310656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62161838563840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4394434 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67780518418432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15150952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18688582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61319790256128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18688582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59364048118368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185:š*†¦0Ñ+ _wandbruntime‚854  _runtime‚854ÇßýgQÒË 2loss/SRqqt3/assignment_loss‚2.2786088675774376 2loss/SRqqt4/assignment_loss‚2.3409694093213953 0loss/FBt1/assignment_loss‚0.6313134816682577 1loss/FBt3/assignment_loss‚0.30825004130700173 /loss/FBt1/detection_loss‚2.7789323292672634 /loss/FBt3/detection_loss‚0.7511992734307735 % _timestamp‚1.7885254706728635e+09 /loss/FRt1/assignment_loss‚6.585745831330617 /loss/FRt1/detection_loss‚3.3911502261956534 ,loss/FRt2/detection_loss‚2.5061856093614 .loss/FRt3/detection_loss‚2.118524030447006 epoch‚0 2loss/SRqqt1/assignment_loss‚2.7456100491377025 /loss/FRt2/assignment_loss‚6.052217276200004 /loss/FRt4/detection_loss‚1.9341830861042528 .loss/FBt4/detection_loss‚4.038276592890422 trainer/global_step‚749 _runtime‚ 854.20819358 2loss/SRqqt2/assignment_loss‚2.1486308785776296 1loss/SRqqt3/detection_loss‚1.5312995250368941 0loss/SRqqt4/detection_loss‚1.687106221704416 %loss/total_loss‚60.27971606028166 _step‚59 /loss/FRt4/assignment_loss‚4.539955938563628 0loss/FBt2/assignment_loss‚0.4982584866975329 1loss/SRqqt1/detection_loss‚3.0592941756431875 /loss/FBt2/detection_loss‚2.7916808620743128 /loss/FRt3/assignment_loss‚2.658536665439606 0loss/FBt4/assignment_loss‚0.4249167541662852 0loss/SRqqt2/detection_loss‚2.478870448138979O<öUÓÏ 2loss/SRqqt3/assignment_loss‚2.2786088675774376 2loss/SRqqt4/assignment_loss‚2.3409694093213953 0loss/FBt1/assignment_loss‚0.6313134816682577 1loss/FBt3/assignment_loss‚0.30825004130700173 /loss/FBt1/detection_loss‚2.7789323292672634 /loss/FBt3/detection_loss‚0.7511992734307735 % _timestamp‚1.7885254706728635e+09 /loss/FRt1/assignment_loss‚6.585745831330617 /loss/FRt1/detection_loss‚3.3911502261956534 ,loss/FRt2/detection_loss‚2.5061856093614 .loss/FRt3/detection_loss‚2.118524030447006 epoch‚0 2loss/SRqqt1/assignment_loss‚2.7456100491377025 /loss/FRt2/assignment_loss‚6.052217276200004 /loss/FRt4/detection_loss‚1.9341830861042528 .loss/FBt4/detection_loss‚4.038276592890422 trainer/global_step‚749 _runtime‚ 854.20819358 2loss/SRqqt2/assignment_loss‚2.1486308785776296 1loss/SRqqt3/detection_loss‚1.5312995250368941 0loss/SRqqt4/detection_loss‚1.687106221704416 %loss/total_loss‚60.27971606028166 _step‚59 /loss/FRt4/assignment_loss‚4.539955938563628 0loss/FBt2/assignment_loss‚0.4982584866975329 1loss/SRqqt1/detection_loss‚3.0592941756431875 /loss/FBt2/detection_loss‚2.7916808620743128 /loss/FRt3/assignment_loss‚2.658536665439606 0loss/FBt4/assignment_loss‚0.4249167541662852 0loss/SRqqt2/detection_loss‚2.478870448138979;öÂú¡“Ô‚(:ˆ Ÿ÷êÔ¤Ôá gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å×Fß Õ‚(:‚ Ÿ÷êÔÐÖÇá disk.sda2.out‚ 88.4970703125 disk.sda1.out‚0 disk.md0.out‚ 84.8359375 disk.sdb.out‚ 88.4970703125 disk.sdb2.in‚0" disk.nvme1n1.out‚ 5386.55859375 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0$ memory_percent‚91.38319492340088) proc.memory.availableMB‚ 1411.77734375 disk.sdb.in‚0 disk.sdb2.out‚ 88.4970703125 disk.md0p1.in‚0 disk.nvme1n1.in‚0 disk.sdb1.out‚0 proc.memory.rssMB‚ 2879.5625 proc.cpu.threads‚17" disk.nvme0n1.out‚ 7209.87109375 disk.sda2.in‚0 disk.md0.in‚0 disk.sda.out‚ 88.4970703125 disk.md0p1.out‚ 84.8359375 disk.sda1.in‚0 disk.rbd2.in‚0* proc.memory.percent‚17.575454711914062 network.sent‚ 1.058624e+06 network.recv‚372208) disk./.usagePercent‚75.88056258140332% disk./.usageGB‚2713.8564643859863 disk.dm-0.in‚0 disk.dm-0.out‚ 12596.4296875 disk.sda.in‚0 cpu‚24.993373630372783 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875Fyé)”Ö‚(:‰ ®÷êÔܸÏà gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nu! ׂ(:û ®÷êÔ¹ŒÆá disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 5386.76171875 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.sdb.in‚0 disk.md0p1.out‚85.6875 proc.memory.rssMB‚ 2879.5625 disk.nvme0n1.out‚ 7522.03125 network.recv‚377653 disk.sda1.out‚0 disk.rbd2.out‚ 0.279296875 disk.sda.in‚0) proc.memory.availableMB‚ 1411.76953125 network.sent‚ 1.109476e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.md0.out‚85.6875 disk.dm-0.out‚12908.79296875 disk.md0p1.in‚0 cpu‚24.994324440596436 disk.sdb1.out‚0 disk.sdb2.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sda2.out‚89.38525390625 disk.sdb.out‚89.38525390625* proc.memory.percent‚17.575454711914062$ disk./.usageGB‚2713.855453491211 disk.sdb2.out‚89.38525390625 disk.sda.out‚89.38525390625# memory_percent‚91.3832426071167( disk./.usagePercent‚75.8805343163623 disk.sdb1.in‚0 disk.rbd2.in‚0¨Üà’Ø‚(:‡ ½÷êÔº‘áÞ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.908' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ق(:ù ½÷êÔöõ‹á) disk./.usagePercent‚75.86578764451221 disk.sdb1.out‚0 disk.dm-0.out‚ 13109.984375 disk.sdb2.out‚ 91.0361328125$ memory_percent‚91.38176441192627) proc.memory.availableMB‚ 1412.01171875 proc.memory.rssMB‚ 2879.5625 cpu‚24.99499430536946 disk.nvme1n1.out‚ 5387.078125 disk.nvme0n1.out‚ 7722.90625 disk.sda2.in‚0 disk.sdb.out‚ 91.0361328125 disk.rbd0.in‚0 disk.sda2.out‚ 91.0361328125 disk.md0.out‚ 87.2578125 disk.sda1.out‚0* proc.memory.percent‚17.575454711914062# disk./.usageGB‚2713.32804107666 disk.md0p1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.115554e+06 disk.sdb.in‚0 network.recv‚382112 disk.sda.in‚0 disk.sda.out‚ 91.0361328125 disk.nvme1n1.in‚0 disk.md0p1.out‚ 87.2578125 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0 disk.sdb2.in‚0:ÀGo”Ú‚(:‰ Ì÷êÔÒ®•ß gpu.0.gpu‚10.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì/‰éÛ‚(:… Ì÷êÔõ‘Äà disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0 disk.sdb.in‚0 disk.md0p1.out‚ 88.3359375# memory_percent‚91.3832426071167 disk.dm-0.out‚ 13268.1484375 disk.sdb.out‚92.18115234375 disk.sdb2.in‚0 disk.sdb2.out‚92.18115234375# proc.memory.rssMB‚ 2879.56640625 cpu‚24.995107639570165 proc.cpu.threads‚17( disk./.usagePercent‚75.8645376897363 disk.sdb1.in‚0 disk.md0.out‚ 88.3359375 disk.nvme1n1.in‚0* proc.memory.percent‚17.575478553771973 network.recv‚386709% disk./.usageGB‚2713.2833366394043 disk.rbd2.out‚ 0.279296875 disk.sda2.in‚0! disk.nvme1n1.out‚ 5387.3203125 network.sent‚ 1.119671e+06 disk.sdb1.out‚0 disk.sda.out‚92.18115234375 disk.rbd0.out‚ 0.279296875 disk.sda1.out‚0 disk.sda2.out‚92.18115234375 disk.md0.in‚0 disk.sda.in‚0 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 7880.828125) proc.memory.availableMB‚ 1411.76953125Aã9¶0Ü+ _wandbruntime‚911  _runtime‚911!â.2ÆÝÀ ' lr-AdamW/pg1‚0.00048362926424350535 ' lr-AdamW/pg2‚0.00048362926424350535 trainer/global_step‚799 % _timestamp‚1.7885255283035228e+09 _runtime‚ 911.839169499 _step‚60c ˆ!ÊÞÄ ' lr-AdamW/pg1‚0.00048362926424350535 ' lr-AdamW/pg2‚0.00048362926424350535 trainer/global_step‚799 % _timestamp‚1.7885255283035228e+09 _runtime‚ 911.839169499 _step‚60<îþ² 0ß+ _wandbruntime‚911  _runtime‚911µáÀ±ÛàÉO _runtime‚ 911.841163348 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19933499 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9659050 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 893668973424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 939847439360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66217066941440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72201577936384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ÕËPbÓ'áÍO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 893668973424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 939847439360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66217066941440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72201577936384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 911.841163348 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19933499 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9659050= 0â+ _wandbruntime‚913  _runtime‚913>TÙ&ãÓM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36117310 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36092436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73232832661504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 % _timestamp‚1.7885255294953065e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66299807500800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9671115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 940818359296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210661376 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64209714092270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898153735680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19933484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚62 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16158952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65400744547328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14361874 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 941029020672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72291779700736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64210639099294 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973Ç5S #ä×M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4690135 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899062953472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 894786246702 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234601472 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 895694214366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19958356 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898153735680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36117310 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36092436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73232832661504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 % _timestamp‚1.7885255294953065e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66299807500800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9671115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 940818359296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210661376 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64209714092270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898153735680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19933484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚62 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16158952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65400744547328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14361874 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 941029020672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72291779700736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64210639099294 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_po³Äi}Ðol.peak‚ 12553587712 Fl#¾Û0å+ _wandbruntime‚913  _runtime‚913ƒ²ÕBæ¼ 0loss/FRt3/assignment_loss‚2.1891406973203025 2loss/SRqqt1/assignment_loss‚3.0228658151626586 /loss/FBt2/assignment_loss‚0.513495611123656 0loss/FBt3/assignment_loss‚0.3100243798810226 _step‚63 /loss/FRt2/assignment_loss‚6.146188718301278 2loss/SRqqt3/assignment_loss‚2.4740831152959304 /loss/FRt1/assignment_loss‚6.440147435223615 loss/FBt4/assignment_loss‚0 1loss/SRqqt1/detection_loss‚3.0428014278411863 0loss/SRqqt2/detection_loss‚2.729047170391789 &loss/total_loss‚60.608361257927314 1loss/SRqqt2/assignment_loss‚2.143924480235135 /loss/FRt1/detection_loss‚3.0858244675177113 .loss/FRt2/detection_loss‚2.850689835018582 .loss/FBt1/detection_loss‚6.175955682992935 -loss/FBt3/detection_loss‚0.90486200764895 /loss/FBt4/detection_loss‚2.4684391021728516 _runtime‚ 913.033251863 1loss/SRqqt4/detection_loss‚1.3703214158525778 % _timestamp‚1.7885255294979405e+09 /loss/FRt4/assignment_loss‚4.054739584093508 epoch‚0 1loss/FBt1/assignment_loss‚0.42594608228318975 /loss/FRt4/detection_loss‚1.8567614255176075 1loss/SRqqt3/detection_loss‚1.7044173619963907 .loss/FBt2/detection_loss‚2.715115399737107 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.2005604214611507 /loss/FRt3/detection_loss‚1.7830096208581738'ƸéFçÀ 0loss/FBt3/assignment_loss‚0.3100243798810226 _step‚63 /loss/FRt2/assignment_loss‚6.146188718301278 2loss/SRqqt3/assignment_loss‚2.4740831152959304 /loss/FRt1/assignment_loss‚6.440147435223615 loss/FBt4/assignment_loss‚0 1loss/SRqqt1/detection_loss‚3.0428014278411863 0loss/SRqqt2/detection_loss‚2.729047170391789 &loss/total_loss‚60.608361257927314 1loss/SRqqt2/assignment_loss‚2.143924480235135 /loss/FRt1/detection_loss‚3.0858244675177113 .loss/FRt2/detection_loss‚2.850689835018582 .loss/FBt1/detection_loss‚6.175955682992935 -loss/FBt3/detection_loss‚0.90486200764895 /loss/FBt4/detection_loss‚2.4684391021728516 _runtime‚ 913.033251863 1loss/SRqqt4/detection_loss‚1.3703214158525778 % _timestamp‚1.7885255294979405e+09 /loss/FRt4/assignment_loss‚4.054739584093508 epoch‚0 1loss/FBt1/assignment_loss‚0.42594608228318975 /loss/FRt4/detection_loss‚1.8567614255176075 1loss/SRqqt3/detection_loss‚1.7044173619963907 .loss/FBt2/detection_loss‚2.715115399737107 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.2005604214611507 /loss/FRt3/detection_loss‚1.7830096208581738 0loss/FRt3/assignment_loss‚2.1891406973203025 2loss/SRqqt1/assignment_loss‚3.0228658151626586 /loss/FBt2/assignment_loss‚0.513495611123656?y3ññ”è‚(:‰ Û÷êÔþ©Äà gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ñ é‚(:€ Û÷êÔéÁ¹á disk.rbd0.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1411.7734375 cpu‚24.99566697628842 disk.sda2.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.md0.out‚ 89.39453125 disk.sdb.out‚ 93.2880859375 disk.sda.out‚ 93.2880859375 proc.cpu.threads‚17% disk./.usageGB‚2713.3555450439453 disk.sda1.out‚0 network.recv‚390878) disk./.usagePercent‚75.86655666694881 disk.nvme0n1.in‚0 disk.dm-0.out‚ 13497.984375 disk.sda.in‚0 disk.rbd0.out‚ 0.279296875# proc.memory.rssMB‚ 2879.56640625 disk.sdb2.out‚ 93.2880859375 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 5459.640625 disk.md0p1.in‚0 network.sent‚ 1.138065e+06 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.out‚ 8038.34375 disk.sda2.out‚ 93.2880859375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0p1.out‚ 89.39453125 disk.sdb2.in‚0* proc.memory.percent‚17.575478553771973 disk.sdb.in‚0 disk.sdb1.in‚0$ memory_percent‚91.38321876525879…“û”ê‚(:‰ ê÷êÔàÕçÞ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F¦zðë‚(:ƒ ê÷êÔåáõà* proc.memory.percent‚17.575478553771973 disk.nvme1n1.in‚0 disk.sda2.in‚0 disk.md0.out‚ 90.39453125 proc.cpu.threads‚17) disk./.usagePercent‚75.87355818425888 disk.sdb.out‚ 94.353515625 network.sent‚ 1.17602e+06 network.recv‚395808 disk.sdb2.in‚0! disk.nvme1n1.out‚ 5527.8828125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 0.279296875 disk.rbd2.in‚0 disk.sda2.out‚ 94.353515625 disk.md0.in‚0 cpu‚24.996288350903686% disk./.usageGB‚2713.6059532165527 disk.sdb2.out‚ 94.353515625 disk.sda.out‚ 94.353515625 disk.md0p1.out‚ 90.39453125 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0" disk.nvme0n1.out‚ 8195.58203125 disk.dm-0.out‚13723.46484375 disk.sda.in‚0$ memory_percent‚91.38343334197998# proc.memory.rssMB‚ 2879.56640625 disk.sdb.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0) proc.memory.availableMB‚ 1411.73828125óRÙ“ì‚(:ˆ ù÷êÔÏÈîà gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05åXší‚(:‡ ù÷êÔˆ¶ªá disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.dm-0.in‚0 disk.sdb2.out‚95.71923828125) disk./.usagePercent‚75.88056236808225 disk.dm-0.out‚ 14076.828125 disk.sdb2.in‚0 disk.sda.out‚95.71923828125 disk.rbd0.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 8405.3359375 disk.md0.in‚0 disk.sdb.out‚95.71923828125) proc.memory.availableMB‚ 1411.92578125$ disk./.usageGB‚2713.856456756592 disk.sda.in‚0 disk.sda1.out‚0 disk.sdb.in‚0* proc.memory.percent‚17.575478553771973 cpu‚24.996839593690815 network.sent‚ 1.182273e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.md0.out‚ 91.69921875 network.recv‚400595! disk.nvme1n1.out‚ 5671.4921875 disk.md0p1.out‚ 91.69921875 disk.rbd2.out‚ 0.279296875 disk.sda2.in‚0# memory_percent‚91.3822889328003# proc.memory.rssMB‚ 2879.56640625 disk.sda2.out‚95.71923828125c¿´’î‚(:‡ ˆøêÔ—†µà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«—v¯ï‚(:† ˆøêÔÎÏ­á disk.sda.out‚ 97.0185546875 disk.md0p1.out‚ 92.92578125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875 disk.nvme0n1.in‚0) disk./.usagePercent‚75.88053388972018 disk.dm-0.in‚0 disk.dm-0.out‚ 14327.296875 disk.sdb.in‚0 disk.sdb2.in‚0 disk.nvme1n1.in‚0 disk.sda2.out‚ 97.0185546875* proc.memory.percent‚17.575478553771973 network.recv‚404775 disk.sdb.out‚ 97.0185546875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.out‚ 92.92578125 proc.cpu.threads‚17 disk.sdb2.out‚ 97.0185546875 disk.sda.in‚0" disk.nvme1n1.out‚ 5764.72265625$ memory_percent‚91.38448238372803 disk.rbd0.out‚ 0.279296875) proc.memory.availableMB‚ 1411.56640625 disk.md0p1.in‚0 disk.md0.in‚0 cpu‚24.997140605580604 network.sent‚ 1.188325e+06$ disk./.usageGB‚2713.855438232422 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.nvme0n1.out‚ 8562.57421875# proc.memory.rssMB‚ 2879.56640625‰GQ0ð+ _wandbruntime‚970  _runtime‚970Á¥éYÄñ¾ _runtime‚ 970.394174648 _step‚64 & lr-AdamW/pg1‚0.0005138939240835244 & lr-AdamW/pg2‚0.0005138939240835244 trainer/global_step‚849 % _timestamp‚1.7885255868585808e+09¶$ÈÈò & lr-AdamW/pg2‚0.0005138939240835244 trainer/global_step‚849 % _timestamp‚1.7885255868585808e+09 _runtime‚ 970.394174648 _step‚64 & lr-AdamW/pg1‚0.0005138939240835244@&ZÚ¾0ó+ _wandbruntime‚970  _runtime‚970¨VWþÑ'ôËO ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950454195988 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38325195 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76712839218688 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 949546228532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 970.395346104 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 953119700480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15242461 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4980799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21153555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69400061749760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67186789978224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10262300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17146794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70353181450240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70354108657664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15243099 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 954028904960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21153555 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76712863158784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 _step‚65 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7885255868596933e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69400079752704 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77711590870528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10262298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68136336206756“Ì¡Õ'õÏO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67186789978224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10262300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17146794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70353181450240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70354108657664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15243099 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 954028904960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21153555 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76712863158784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 _step‚65 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7885255868596933e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69400079752704 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77711590870528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10262298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68136336206756 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950454195988 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38325195 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76712839218688 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 949546228532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 970.395346104 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 953119700480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15242461 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4980799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21153555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69400061749760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992AXÊÓ 0ö+ _wandbruntime‚971  _runtime‚971­}É¡÷ÕM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70436849217024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70436849217024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21203258 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/activš'¡Q:e_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 955150378496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 954241160704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10274365 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15260896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38345338 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21178386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10274363 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4986531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 999709068288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67265807572768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4985910 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973óh ß&øÙM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69481698838528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77803006495744 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 950663501810 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38345338 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69481698838528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚ 971.486960595 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69481680835584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21203258 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 955150378496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 954241160704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10274365 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15260896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38345338 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21178386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10274363 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4986531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 999709068288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67265807572768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4985910 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70436849217024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70436849217024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010116329984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10865548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1057950310912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73481034043904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚73481016040960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 % _timestamp‚1.7885256454563925e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82282098692096 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82281861980160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40578097 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71137669705424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1028.992355883 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 _step‚6Ëš‡ÂÁ9 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5277277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74491150373888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22423303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81224124441088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74490223166464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 trainer/global_step‚899 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81224148381184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1005423483640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009207125504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22398457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ¼iÞ'‰ØO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40578097 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71137669705424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1028.992355883 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 _step‚69 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5277277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74491150373888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22423303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81224124441088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74490223166464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 trainer/global_step‚899 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81224148381184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1005423483640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009207125504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22398457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72144018195880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22398457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚73481034043904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22423303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18154794 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1006331451096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010116329984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10865548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1057950310912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73481034043904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚73481016040960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 % _timestamp‚1.7885256454563925e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82282098692096 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82281861980160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615EÐUMž2Š- _wandbruntime‚1030  _runtime‚1030¿{Aä&‹ÞM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚70 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1010328585728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1006540756918 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74572963712512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72224153063910 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40623114 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245087232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71216687299968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1059139910144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74573890933248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73562653129728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886è.˜á( ŒâM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72224153063910 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40623114 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245087232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71216687299968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1059139910144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74573890933248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73562653129728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚70 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1010328585728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1006540756918 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74572963712512F+”'„2- _wandbruntime‚1030  _runtime‚1030¹Ž6¯XŽÒ trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚2.6840531701842942 1loss/FBt4/assignment_loss‚0.20545423030853271 1loss/SRqqt3/detection_loss‚1.2869156645217412 /loss/FRt2/assignment_loss‚5.767220576604207 /loss/FRt4/assignment_loss‚3.794651758670807 2loss/SRqqt1/assignment_loss‚3.0876090717315674 /loss/FRt2/detection_loss‚2.6398290495077767 _runtime‚1030.185923117 _step‚71 .loss/FRt4/detection_loss‚1.901660024694034 1loss/SRqqt2/detection_loss‚3.8399451772371926 /loss/FBt3/detection_loss‚0.9023011731372661 epoch‚0 0loss/FBt2/assignment_loss‚0.6387753823519629 /loss/FRt3/detection_loss‚1.6890100895381364 .loss/FBt1/detection_loss‚3.088018218676249 /loss/FBt2/detection_loss‚2.1873128621474556 %loss/total_loss‚59.39475553698186 /loss/FRt1/detection_loss‚3.3760755333033474 /loss/FRt1/assignment_loss‚6.037109938534823 /loss/FRt3/assignment_loss‚2.490887399579658 2loss/SRqqt3/assignment_loss‚1.6104830549636358 0loss/FBt1/assignment_loss‚0.6204702511895448 1loss/FBt3/assignment_loss‚0.29468767757077136 % _timestamp‚1.7885256466505215e+09 2loss/SRqqt4/assignment_loss‚1.8745664754245854 1loss/SRqqt1/detection_loss‚2.9395363306999207 1loss/SRqqt4/detection_loss‚1.5047672519200965 .loss/FBt4/detection_loss‚4.933415174484253æÃI\Ö 2loss/SRqqt1/assignment_loss‚3.0876090717315674 /loss/FRt2/detection_loss‚2.6398290495077767 _runtime‚1030.185923117 _step‚71 .loss/FRt4/detection_loss‚1.901660024694034 1loss/SRqqt2/detection_loss‚3.8399451772371926 /loss/FBt3/detection_loss‚0.9023011731372661 epoch‚0 0loss/FBt2/assignment_loss‚0.6387753823519629 /loss/FRt3/detection_loss‚1.6890100895381364 .loss/FBt1/detection_loss‚3.088018218676249 /loss/FBt2/detection_loss‚2.1873128621474556 %loss/total_loss‚59.39475553698186 /loss/FRt1/detection_loss‚3.3760755333033474 /loss/FRt1/assignment_loss‚6.037109938534823 /loss/FRt3/assignment_loss‚2.490887399579658 2loss/SRqqt3/assignment_loss‚1.6104830549636358 0loss/FBt1/assignment_loss‚0.6204702511895448 1loss/FBt3/assignment_loss‚0.29468767757077136 % _timestamp‚1.7885256466505215e+09 2loss/SRqqt4/assignment_loss‚1.8745664754245854 1loss/SRqqt1/detection_loss‚2.9395363306999207 1loss/SRqqt4/detection_loss‚1.5047672519200965 .loss/FBt4/detection_loss‚4.933415174484253 trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚2.6840531701842942 1loss/FBt4/assignment_loss‚0.20545423030853271 1loss/SRqqt3/detection_loss‚1.2869156645217412 /loss/FRt2/assignment_loss‚5.767220576604207 /loss/FRt4/assignment_loss‚3.794651758670807GgPÏŽ”‚(:‰ ÓøêÔÞ¦îà gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UŽÈé‘‚(:ƒ ÓøêÔ€åÔá! disk.nvme0n1.out‚ 9191.8359375 disk.sdb.in‚0 disk.sdb2.out‚ 103.220703125* proc.memory.percent‚17.575478553771973 disk.md0p1.out‚ 98.80078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.md0.in‚0 disk.sda2.out‚ 103.220703125 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 6046.7734375# disk./.usageGB‚2713.60595703125 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚ 15238.609375 disk.sdb2.in‚0 network.sent‚ 1.269795e+06 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.sda.in‚0 disk.sda.out‚ 103.220703125 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 103.220703125# proc.memory.rssMB‚ 2879.56640625 cpu‚24.999217116699693) disk./.usagePercent‚75.87355829091942 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875 disk.md0.out‚ 98.80078125$ memory_percent‚91.38338565826416) proc.memory.availableMB‚ 1411.74609375 network.recv‚425772 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0ʈ!ž’‚(:“ âøêÔðÿ¿à gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ ’G“‚(:ˆ âøêÔÜÚöà disk.sdb2.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 99.69921875# proc.memory.rssMB‚ 2879.56640625 proc.cpu.threads‚17 disk.rbd2.out‚ 0.279296875! disk.sda2.out‚104.17529296875 disk.md0.in‚0! disk.sdb2.out‚104.17529296875 disk.nvme1n1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.575478553771973 network.recv‚431125$ disk./.usageGB‚2713.856460571289$ memory_percent‚91.38336181640625) disk./.usagePercent‚75.88056247474279 disk.sdb.out‚104.17529296875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.307852e+06" disk.nvme0n1.out‚ 9453.89453125 disk.md0.out‚ 99.69921875 disk.dm-0.in‚0 disk.nvme1n1.out‚ 6201.859375 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.sdb1.in‚0# proc.memory.availableMB‚1411.75 disk.dm-0.out‚15655.75390625 disk.sdb.in‚0 disk.sda.in‚0 disk.sda.out‚104.17529296875 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 cpu‚24.999496053187173X¸4W“”‚(:ˆ ñøêÔ¡ÎÆà gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.294' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.0735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0® n~•‚(:‰ ñøêÔ§ÙÂá network.sent‚ 1.311957e+06 disk.sdb.in‚0 disk.sdb.out‚105.26123046875 disk.md0p1.in‚0 disk.md0p1.out‚ 100.734375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0" disk.nvme0n1.out‚ 9558.82421875$ disk./.usageGB‚2713.855442047119 disk.sdb2.in‚0! disk.sdb2.out‚105.26123046875 disk.sda.in‚0 disk.nvme1n1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875( proc.memory.availableMB‚ 1411.7421875( disk./.usagePercent‚75.8805339963807 disk.sda.out‚105.26123046875 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0! disk.sda2.out‚105.26123046875* proc.memory.percent‚17.575478553771973 proc.cpu.threads‚17$ memory_percent‚91.38340950012207 disk.md0.in‚0 disk.md0.out‚ 100.734375 disk.dm-0.out‚15838.41796875 disk.nvme1n1.out‚ 6279.59375 disk.sda1.out‚0 network.recv‚435881# proc.memory.rssMB‚ 2879.56640625 cpu‚25.000163238942026¥¨ 9’–‚(:‡ €ùêԙخà gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zz™[—‚(:Š €ùêÔ—é†á disk.sdb1.in‚0 disk.sda1.out‚0 network.sent‚ 1.320184e+06 disk.sdb1.out‚0# proc.memory.rssMB‚ 2879.56640625 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 9716.078125 disk.sda.in‚0 disk.sda.out‚106.5224609375 disk.md0p1.in‚0# memory_percent‚91.3825511932373 network.recv‚440310 disk.md0.out‚ 101.93359375 disk.sdb.out‚106.5224609375 disk.sdb2.in‚0 disk.md0p1.out‚ 101.93359375 disk.sda1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 cpu‚25.00040967475522) disk./.usagePercent‚75.86579415080467% disk./.usageGB‚2713.3282737731934 disk.rbd2.out‚ 0.279296875 disk.sda2.out‚106.5224609375 disk.sdb2.out‚106.5224609375 disk.nvme1n1.in‚0* proc.memory.percent‚17.575478553771973 disk.rbd0.out‚ 0.279296875 proc.cpu.threads‚17 disk.sda2.in‚0" disk.nvme1n1.out‚ 6486.30078125( proc.memory.availableMB‚ 1411.8828125 disk.dm-0.in‚0 disk.dm-0.out‚16202.37890625 disk.sdb.in‚0¤° 2˜- _wandbruntime‚1087  _runtime‚1087µZÅ™¿ & lr-AdamW/pg1‚0.0005744232437635626 & lr-AdamW/pg2‚0.0005744232437635626 trainer/global_step‚949 % _timestamp‚1.7885257041243384e+09 _runtime‚1087.660858818 _step‚72³_×Òɚà & lr-AdamW/pg1‚0.0005744232437635626 & lr-AdamW/pg2‚0.0005744232437635626 trainer/global_step‚949 % _timestamp‚1.7885257041243384e+09 _runtime‚1087.660858818 _step‚72HÐÇ’K2›- _wandbruntime‚1087  _runtime‚1087~DÚ'œÔO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _runtime‚1087.661022079 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85735409663488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885257041251872e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23643359 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1116677530624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1065294550528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85735433603584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1116896593920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75088566471984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19162794 _step‚73 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78627264882688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162794 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19162792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1065294550528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23668205 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11468800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23643359 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77561970332160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86852330197504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78628192090112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1061300738748 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77561970332160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78628192090112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1066203755008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77561988335104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0O+²âÞ'ØO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78628192090112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1061300738748 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77561970332160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78628192090112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1066203755008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77561988335104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _runtime‚1087.661022079 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85735409663488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885257041251872e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23643359 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1116677530624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1065294550528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85735433603584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1116896593920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75088566471984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19162794 _step‚73 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78627264882688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162794 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19162792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1065294550528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23668205 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11468800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23643359 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77561970332160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86852330197504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615Iü3x2ž- _wandbruntime‚1088  _runtime‚1088&GæQŸÞM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1062418012026 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42851142 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1066416010752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75167584066528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_baÕÕ¦n“!tch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚1088.679753052 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42876016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1063325979690 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77643607420928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19182954 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19182954 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76229985039194 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75167567027168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚949 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77643589417984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86943739535872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11480863 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221147136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1067325228544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5578694 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615oH”iè& âM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1062418012026 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42851142 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1066416010752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75167584066528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚1088.679753052 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42876016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1063325979690 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77643607420928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19182954 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19182954 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76229985039194 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75167567027168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚949 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77643589417984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86943739535872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11480863 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221147136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1067325228544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5578694 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262Jßý%±2¡- _wandbruntime‚1088  _runtime‚1088‹®aS¢Í .loss/FRt1/detection_loss‚3.362478557087126 /loss/FBt3/detection_loss‚0.7437169626624381 _runtime‚1088.680798219 0loss/FBt2/assignment_loss‚0.9560276667277018 /loss/FRt2/assignment_loss‚5.594558700438468 /loss/FRt3/assignment_loss‚2.221235813839095 0loss/FBt1/assignment_loss‚0.5665695204801383 .loss/FRt4/detection_loss‚2.096741756986058 %loss/total_loss‚58.40503716767347 _step‚75 0loss/FRt4/assignment_loss‚4.0906560988653276 0loss/FBt3/assignment_loss‚0.2509749982052262 /loss/FRt3/detection_loss‚1.6321695380445038 /loss/FBt4/detection_loss‚3.4844547510147095 1loss/SRqqt1/detection_loss‚2.9533192863066993 0loss/SRqqt3/detection_loss‚1.213958884589374 0loss/FBt4/assignment_loss‚0.6290023028850555 /loss/FRt2/detection_loss‚2.7567687496062248 0loss/SRqqt4/detection_loss‚1.443562531773048 epoch‚0 trainer/global_step‚949 /loss/FRt1/assignment_loss‚5.407587778000605 2loss/SRqqt2/assignment_loss‚2.4521764178533814 2loss/SRqqt4/assignment_loss‚1.9670087735675559 1loss/SRqqt2/detection_loss‚2.7530596530115283 /loss/FBt2/detection_loss‚4.6784921288490295 $ _timestamp‚1.788525705145524e+09 1loss/SRqqt1/assignment_loss‚2.906123585999012 2loss/SRqqt3/assignment_loss‚1.3613890083506703 .loss/FBt1/detection_loss‚2.883003702530494+kTùW£Ñ /loss/FRt3/assignment_loss‚2.221235813839095 0loss/FBt1/assignment_loss‚0.5665695204801383 .loss/FRt4/detection_loss‚2.096741756986058 %loss/total_loss‚58.40503716767347 _step‚75 0loss/FRt4/assignment_loss‚4.0906560988653276 0loss/FBt3/assignment_loss‚0.2509749982052262 /loss/FRt3/detection_loss‚1.6321695380445038 /loss/FBt4/detection_loss‚3.4844547510147095 1loss/SRqqt1/detection_loss‚2.9533192863066993 0loss/SRqqt3/detection_loss‚1.213958884589374 0loss/FBt4/assignment_loss‚0.6290023028850555 /loss/FRt2/detection_loss‚2.7567687496062248 0loss/SRqqt4/detection_loss‚1.443562531773048 epoch‚0 trainer/global_step‚949 /loss/FRt1/assignment_loss‚5.407587778000605 2loss/SRqqt2/assignment_loss‚2.4521764178533814 2loss/SRqqt4/assignment_loss‚1.9670087735675559 1loss/SRqqt2/detection_loss‚2.7530596530115283 /loss/FBt2/detection_loss‚4.6784921288490295 $ _timestamp‚1.788525705145524e+09 1loss/SRqqt1/assignment_loss‚2.906123585999012 2loss/SRqqt3/assignment_loss‚1.3613890083506703 .loss/FBt1/detection_loss‚2.883003702530494 .loss/FRt1/detection_loss‚3.362478557087126 /loss/FBt3/detection_loss‚0.7437169626624381 _runtime‚1088.680798219 0loss/FBt2/assignment_loss‚0.9560276667277018 /loss/FRt2/assignment_loss‚5.594558700438468Kº©”¤‚(:‰ ùêÔ¹ûìà gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04$Û—¥‚(:Œ ùêÔ·à¡á disk.sda.out‚108.3427734375* proc.memory.percent‚17.575478553771973 disk.sda.in‚0) disk./.usagePercent‚75.86460509919259 disk.nvme0n1.in‚0 disk.sdb2.out‚108.3427734375 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.sda1.out‚0 disk.sda2.out‚108.3427734375 disk.sdb.in‚0 disk.sdb.out‚108.3427734375 disk.nvme1n1.in‚0 cpu‚25.00106451559417 network.recv‚444274 disk.dm-0.in‚0 disk.sdb2.in‚0) proc.memory.availableMB‚ 1411.78515625 network.sent‚ 1.336255e+06 disk.sda1.in‚0" disk.nvme0n1.out‚ 9873.96484375 disk.md0.in‚0 disk.md0.out‚ 103.68359375 disk.md0p1.in‚0 disk.md0p1.out‚ 103.68359375# proc.memory.rssMB‚ 2879.56640625$ disk./.usageGB‚2713.285747528076 disk.rbd0.in‚0 disk.rbd2.out‚ 0.279296875 disk.dm-0.out‚16404.00390625 disk.sdb1.in‚0! disk.nvme1n1.out‚ 6530.0390625$ memory_percent‚91.38314723968506 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0†p¶{’¦‚(:‡ žùêÔÇÂà gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öà«¢§‚(:‘ žùêÔ†ë£á disk.rbd0.out‚ 0.279296875 network.sent‚ 1.37221e+06 disk.rbd2.out‚ 0.279296875# disk.nvme0n1.out‚10102.95703125 disk.sda2.in‚0 disk.sdb.in‚0 disk.md0p1.out‚ 104.69921875 network.recv‚449645 disk.md0.in‚0 disk.dm-0.out‚ 16633.2109375 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1411.61328125 disk.dm-0.in‚0! disk.sdb2.out‚109.42236328125 disk.sda.out‚109.42236328125 disk.md0p1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2713.3550186157227! disk.sda2.out‚109.42236328125 disk.sdb2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚75.86654194779538 disk.rbd2.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0 cpu‚25.001490680795428 disk.sdb.out‚109.42236328125 disk.sda.in‚0* proc.memory.percent‚17.575478553771973 disk.md0.out‚ 104.69921875" disk.nvme1n1.out‚ 6530.25390625 disk.sda1.out‚0# memory_percent‚91.3841962814331# proc.memory.rssMB‚ 2879.56640625:"ù]¨‚(:… ­ùêÔ€½à gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.52' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±HG©‚(:‹ ­ùêÔ­ú¼á disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.sdb.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1411.47265625 proc.cpu.threads‚17 disk.md0.out‚ 105.671875 disk.dm-0.in‚0 disk.md0p1.out‚ 105.671875 disk.sdb1.in‚0# proc.memory.rssMB‚ 2879.56640625% disk./.usageGB‚2713.4804458618164 disk.dm-0.out‚16897.09765625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.279296875* proc.memory.percent‚17.575478553771973 disk.sda.out‚110.4482421875 disk.md0p1.in‚0 disk.sda1.in‚0 cpu‚25.001897680240496 network.sent‚ 1.380434e+06 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 6530.5390625 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚110.4482421875) disk./.usagePercent‚75.87004894609154 disk.sdb2.in‚0 disk.sdb2.out‚110.4482421875 disk.sda2.in‚0$ memory_percent‚91.38505458831787# disk.nvme0n1.out‚10366.55859375 network.recv‚454298 disk.md0.in‚0 disk.sdb.out‚110.4482421875c–¬ ”ª‚(:‰ ¼ùêÔÒžíà gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ö#í«‚(:Š ¼ùêÔÜÓá network.sent‚ 1.386488e+06$ disk./.usageGB‚2713.605178833008 disk.sdb1.in‚0 disk.dm-0.out‚ 17131.9453125) disk./.usagePercent‚75.87353653217086 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 10601.2578125! disk.sdb2.out‚111.88916015625) proc.memory.availableMB‚ 1411.83984375# proc.memory.rssMB‚ 2879.56640625 disk.md0.in‚0 disk.dm-0.in‚0 disk.sdb.in‚0 disk.sda.in‚0 disk.nvme1n1.in‚0 disk.rbd2.out‚ 0.279296875! disk.sda2.out‚111.88916015625 disk.sdb.out‚111.88916015625 disk.sdb2.in‚0 disk.md0p1.out‚ 107.0390625$ memory_percent‚91.38281345367432* proc.memory.percent‚17.575478553771973 disk.nvme1n1.out‚ 6530.6875 network.recv‚458681 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 disk.sda.out‚111.88916015625 disk.md0p1.in‚0 cpu‚25.00229467970696 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.md0.out‚ 107.0390625 proc.cpu.threads‚17 #}¼2¬- _wandbruntime‚1146  _runtime‚1146S†o[Å­¿ % _timestamp‚1.7885257627114432e+09 _runtime‚1146.246991536 _step‚76 & lr-AdamW/pg1‚0.0006046879036035818 & lr-AdamW/pg2‚0.0006046879036035818 trainer/global_step‚999‡ïwɮà & lr-AdamW/pg2‚0.0006046879036035818 trainer/global_step‚999 % _timestamp‚1.7885257627114432e+09 _runtime‚1146.246991536 _step‚76 & lr-AdamW/pg1‚0.0006046879036035818LTäö2¯- _wandbruntime‚1146  _runtime‚1146é·[‘°ÔO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45059053 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45083901 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80157532160496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 % _timestamp‚1.7885257627125936e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1118085961312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91422383030272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1175688144384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20170794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24913107 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1175900916224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79039446199184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17940611 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79039429159824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81642924623360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24888261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5869216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90246694885888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81642942626304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20170794 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1121381975552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81642942626304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1122291180032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12072050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1117177993856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5868563 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _step‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1121381975552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24888261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1bXLRÞ'±ØO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24888261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5869216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90246694885888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81642942626304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20170794 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1121381975552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81642942626304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1122291180032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12072050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1117177993856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5868563 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _step‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1121381975552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24888261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45059053 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45083901 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80157532160496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 % _timestamp‚1.7885257627125936e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1118085961312 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91422383030272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1175688144384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20170794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24913107 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1175900916224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79039446199184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17940611 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79039429159824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81642924623360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90336920590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1177089595904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79118446754368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1118295267134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1119203234798 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82847974365696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91514034126336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12084113 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1122503435776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12084115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17959117 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45104044 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20190954 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24937964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82847047144960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1263 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90336944530432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82847974365696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80236742021502 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5875002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24913092 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81724543709184ä º]¡´áM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1122503435776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12084115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17959117 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45104044 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20190954 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24937964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82847047144960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1263 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90336944530432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82847974365696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80236742021502 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5875002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24913092 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81724543709184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90336920590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1177089595904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79118446754368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ (½ì×F 14134971392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1118295267134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1119203234798 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82847974365696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91514034126336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12084113N«¶2µ- _wandbruntime‚1147  _runtime‚1147€ž!»W¶Ñ /loss/FRt1/detection_loss‚3.4454102136872033 /loss/FRt1/assignment_loss‚5.093257177959789 /loss/FRt4/assignment_loss‚3.842635423772865 2loss/SRqqt2/assignment_loss‚2.5135137513279915 1loss/SRqqt1/detection_loss‚2.6121272387043124 .loss/FBt4/detection_loss‚5.961799184481303 trainer/global_step‚999 1loss/FBt4/assignment_loss‚0.39394111434618634 /loss/FRt2/assignment_loss‚5.143543852700128 1loss/SRqqt1/assignment_loss‚2.614912365713427 /loss/FRt3/detection_loss‚1.9244836369213068 1loss/SRqqt2/detection_loss‚3.1183034852147102 1loss/SRqqt4/detection_loss‚1.3944337717307511 /loss/FBt1/detection_loss‚2.9066160917282104 /loss/FBt3/detection_loss‚0.6847552369391481 /loss/FRt4/detection_loss‚1.9910098227361839 /loss/FBt2/detection_loss‚2.5348887899342705 $loss/total_loss‚57.1495907190329 epoch‚0 /loss/FRt2/detection_loss‚2.6850230075694896 2loss/SRqqt3/assignment_loss‚1.1879963869036272 _runtime‚1147.364136644 _step‚79 0loss/FBt1/assignment_loss‚0.7469210013514385 1loss/SRqqt3/detection_loss‚1.5301615084553588 $ _timestamp‚1.788525763828819e+09 0loss/FBt3/assignment_loss‚0.4448567662643263 2loss/SRqqt4/assignment_loss‚1.7823310928639544 0loss/FBt2/assignment_loss‚0.7272019162363208 0loss/FRt3/assignment_loss‚1.8694678814905994DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26133163 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1234894667264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885258210870557e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47336803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177469400576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230420480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1235101147648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86901348315136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1178378605056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21178792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚81 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12675300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85723878914560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84164289142804 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26133163 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47311955 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21178792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26158009 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1173055248964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18839487 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86902275522560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177469400576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178794 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12675298 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84163364135988 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚1049 _runtime‚1204.623019864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26158009 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0™ê¼ß'ÅÙO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177469400576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230420480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1235101147648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86901348315136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1178378605056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21178792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚81 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12675300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85723878914560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84164289142804 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26133163 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47311955 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21178792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26158009 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1173055248964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18839487 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86902275522560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177469400576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178794 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12675298 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84163364135988 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚1049 _runtime‚1204.623019864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26158009 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85723878914560 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26133163 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1234894667264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885258210870557e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47336803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Q(2F2Æ- _wandbruntime‚1205  _runtime‚1205 <êöÂÇßM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 D;DeviceStatsMonitor.on_t?„Y#%rain_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1178590860800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85805498000384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1175080489906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚1049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1178590860800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96084272939520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84244424010834 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86985016081920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26157994 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1236067126784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26182866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736‘Ö3é&ÈãM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12687365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12687363 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1178590860800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96084272939520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84244424010834 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86985016081920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26157994 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1236067126784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26182866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1178590860800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85805498000384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1175080489906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚1049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91039317238784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49564857 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89804833205760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚85 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 MGDeviceStatsMonÚ ¼Àitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27378065 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19740451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89804833205760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1294115831808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885258795692334e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214868992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86941188614224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1229840471528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88170121118296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13278550 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27378065 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91039317238784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22186792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100563619971584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89804851208704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6461243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100563381162496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13278548 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99269265330688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22186792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234466030080O'á'ÙÛO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234466030080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _runtime‚1263.105070911 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91039317238784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49564857 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89804833205760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚85 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27378065 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19740451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89804833205760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1294115831808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885258795692334e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214868992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86941188614224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1229840471528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88170121118296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13278550 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27378065 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91039317238784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22186792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100563619971584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89804851208704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6461243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100563381162496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13278548 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99269265330688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22186792UT³ž2Ú- _wandbruntime‚1264  _runtime‚1264ÚEí)ç&ÛáM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100655033576448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99359514975232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885258806137016e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚86 _runtime‚1264.150010294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91122057798144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49634722 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6467177 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89886452291584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1295299551232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89886452291584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13290613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1230957745014 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99359491035136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27427768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Õâ]#ÜåM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91122057798144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49634722 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6467177 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89886452291584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1295299551232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89886452291584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13290613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1230957745014 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99359491035136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27427768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100655033576448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceS;ª¨ÝÈ tatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99359514975232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885258806137016e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚86 _runtime‚1264.150010294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722V¯rô‡2Ý- _wandbruntime‚1264  _runtime‚1264ë¶UÞÏ /loss/FRt2/assignment_loss‚6.049309765851056 0loss/FRt3/assignment_loss‚2.1911482141728986 /loss/FRt1/detection_loss‚3.0622511964577894 /loss/FRt1/assignment_loss‚5.330216095997737 1loss/SRqqt3/assignment_loss‚1.299441046267748 1loss/FBt3/assignment_loss‚0.30282559878495957 /loss/FRt2/detection_loss‚2.9429129891925387 /loss/FRt4/detection_loss‚1.9198232013732195 /loss/FBt1/detection_loss‚3.1358747363090513 &loss/total_loss‚56.194028268506756 epoch‚0 2loss/SRqqt4/assignment_loss‚1.6878018162904247 1loss/SRqqt1/detection_loss‚2.3674426449311747 /loss/FBt3/detection_loss‚0.6994453360953131 trainer/global_step‚1099 _runtime‚1264.151661601 _step‚87 1loss/SRqqt1/assignment_loss‚2.577187115678916 2loss/SRqqt2/assignment_loss‚2.7380133476108313 /loss/FBt4/assignment_loss‚1.478358268737793 .loss/FRt3/detection_loss‚1.613635128527357 -loss/FBt2/detection_loss‚2.52839232981205 .loss/FBt4/detection_loss‚4.093169689178467 /loss/FRt4/assignment_loss‚3.562445655465126 0loss/FBt2/assignment_loss‚0.6442210553708719 0loss/SRqqt2/detection_loss‚2.640457520261407 1loss/SRqqt4/detection_loss‚1.2814751321109392 0loss/FBt1/assignment_loss‚0.6563913363381289 % _timestamp‚1.7885258806163797e+09 1loss/SRqqt3/detection_loss‚1.3917890476909551ãXcDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1290553455104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885259377367442e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93885805499904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28647813 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6758101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1290553455104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103780574493184 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92177803107420 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92176878100604 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28647813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95176358955008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90892068341424 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚1321.272516496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103780574493184 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92177803107420 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92176878100604 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28647813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95176358955008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90892068341424 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚1321.272516496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1290553455104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885259377367442e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93885805499904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28647813 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6758101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1290553455104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 1322.29463711 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93967424585728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51862750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1290765710848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92257937975450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103870776257536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93967406582784 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105225188743680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212758528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚1149 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95258172293632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93967406582784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93967424585728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236698624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13893863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13893865 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1354412486144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973“»ßì&ðæM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 1322.29463711 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93967424585728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51862750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1290765710848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92257937975450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103870776257536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93967406582784 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105225188743680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212758528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚1149 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95258172293632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93967406582784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93967424585728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236698624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13893863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13893865 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1354412486144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1379.541971304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54070661 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24202792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202794 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1341594981744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54095509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109704404581888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.l‚râ6–arge_pool.allocated‚97966759791104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99313400671232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21540371 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1412323705856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99313400671232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1345731675648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109704159481344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29867869 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚93 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1199 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1345731675648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97966741788160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29867869 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94842948068624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0æKRÖã'ÿÝO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1341594981744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54095509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109704404581888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97966759791104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99313400671232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21540371 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1412323705856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99313400671232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1345731675648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109704159481344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29867869 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚93 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1199 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1345731675648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97966741788160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29867869 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94842948068624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21539693 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1379.541971304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54070661 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24202792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202794 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0].10“€‚(:ˆ ¬ûêÔ×é±à gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è‡ä¥‚(:‘ ¬ûêÔ®¾Ëá disk.sdb2.in‚0! disk.sdb2.out‚132.15673828125( proc.memory.availableMB‚ 1411.3671875! proc.memory.rssMB‚ 2879.578125% disk./.usageGB‚2713.3289909362793 disk.sda.out‚132.15673828125* proc.memory.percent‚17.575550079345703 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 network.recv‚530310 disk.sdb.in‚0 disk.sdb.out‚132.15673828125 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.86581420298472 disk.dm-0.out‚20874.07421875 disk.sdb1.in‚0" disk.nvme0n1.out‚ 13276.9765625 disk.md0.in‚0$ memory_percent‚91.38569831848145 disk.md0p1.in‚0 disk.rbd0.out‚ 0.62109375 cpu‚25.006890684439508 disk.dm-0.in‚0 disk.sda.in‚0 disk.md0p1.out‚ 126.33984375 disk.rbd2.out‚ 0.57080078125 network.sent‚ 1.662129e+06 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 7605.72265625! disk.sda2.out‚132.15673828125 disk.md0.out‚ 126.33984375íL^ÿ2‚- _wandbruntime‚1380  _runtime‚1380ø,~é&ƒãM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 % _timestamp‚1.7885259970995119e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228310016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99396141230592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7061250 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94921982702528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94921965663168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1342712255230 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109795818602496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29917572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1413528812544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98048378876928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99395214009856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 _runtime‚1380.636812569 _step‚94 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98048360873984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14497113 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24222954 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98048360873984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚1199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108382085420032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24222952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96264694957758 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29892700 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54140526 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1346853135872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14497115 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7060589 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392¯NŒ„çM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98048360873984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚1199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108382085420032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24222952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96264694957758 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29892700 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54140526 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1346853135872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14497115 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7060589 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.cur'¤¨Äarent‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 % _timestamp‚1.7885259970995119e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228310016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99396141230592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7061250 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94921982702528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94921965663168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1342712255230 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109795818602496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29917572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1413528812544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98048378876928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99395214009856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 _runtime‚1380.636812569 _step‚94 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98048360873984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14497113 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24222954 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0^4å2…- _wandbruntime‚1380  _runtime‚1380¯´¬U†Ï %loss/total_loss‚50.81436382757906 1loss/SRqqt1/assignment_loss‚2.718878080447515 2loss/SRqqt3/assignment_loss‚1.2182952215986662 0loss/FBt2/assignment_loss‚0.5262369976507232 /loss/FBt4/assignment_loss‚0.313575804233551 1loss/SRqqt1/detection_loss‚2.5812215824921925 /loss/FRt2/assignment_loss‚4.949531866156536 0loss/FRt4/assignment_loss‚3.2788997164794376 /loss/FBt1/assignment_loss‚0.592530429091615 0loss/FBt3/assignment_loss‚0.2404265759722407 .loss/FRt3/detection_loss‚1.689720494524827 /loss/SRqqt2/detection_loss‚2.35757073657266 /loss/FBt2/detection_loss‚2.3364114399467195 1loss/SRqqt3/detection_loss‚1.2468255360106952 % _timestamp‚1.7885259971028156e+09 _step‚95 2loss/SRqqt2/assignment_loss‚2.0633395157497505 2loss/SRqqt4/assignment_loss‚1.5549932815134526 /loss/FRt2/detection_loss‚2.5644012119459068 .loss/FRt4/detection_loss‚2.127600143530539 .loss/FBt4/detection_loss‚2.679920017719269 /loss/FBt3/detection_loss‚0.6505192775742861 0loss/FRt1/assignment_loss‚5.2262165169966845 1loss/SRqqt4/detection_loss‚1.4848772577941418 /loss/FBt1/detection_loss‚2.8439371585845947 0loss/FRt3/assignment_loss‚1.9849959034768363 /loss/FRt1/detection_loss‚3.5834390615162097 trainer/global_step‚1199 epoch‚0 _runtime‚1380.638255795÷wÉY‡Ó trainer/global_step‚1199 epoch‚0 _runtime‚1380.638255795 %loss/total_loss‚50.81436382757906 1loss/SRqqt1/assignment_loss‚2.718878080447515 2loss/SRqqt3/assignment_loss‚1.2182952215986662 0loss/FBt2/assignment_loss‚0.5262369976507232 /loss/FBt4/assignment_loss‚0.313575804233551 1loss/SRqqt1/detection_loss‚2.5812215824921925 /loss/FRt2/assignment_loss‚4.949531866156536 0loss/FRt4/assignment_loss‚3.2788997164794376 /loss/FBt1/assignment_loss‚0.592530429091615 0loss/FBt3/assignment_loss‚0.2404265759722407 .loss/FRt3/detection_loss‚1.689720494524827 /loss/SRqqt2/detection_loss‚2.35757073657266 /loss/FBt2/detection_loss‚2.3364114399467195 1loss/SRqqt3/detection_loss‚1.2468255360106952 % _timestamp‚1.7885259971028156e+09 _step‚95 2loss/SRqqt2/assignment_loss‚2.0633395157497505 2loss/SRqqt4/assignment_loss‚1.5549932815134526 /loss/FRt2/detection_loss‚2.5644012119459068 .loss/FRt4/detection_loss‚2.127600143530539 .loss/FBt4/detection_loss‚2.679920017719269 /loss/FBt3/detection_loss‚0.6505192775742861 0loss/FRt1/assignment_loss‚5.2262165169966845 1loss/SRqqt4/detection_loss‚1.4848772577941418 /loss/FBt1/detection_loss‚2.8439371585845947 0loss/FRt3/assignment_loss‚1.9849959034768363 /loss/FRt1/detection_loss‚3.5834390615162097_„σ’ˆ‚(:‡ »ûêÔò‰ñà gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01…4(: »ûêÔ×ü¨á) disk./.usagePercent‚75.86455838187952 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sdb2.in‚0 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2978.0234375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.sdb.out‚134.01904296875 disk.rbd0.in‚0 network.recv‚535382 disk.nvme0n1.in‚0 disk.sdb.in‚0 cpu‚25.018103467843268 disk.sdb1.out‚0 disk.sda1.out‚0! disk.sda2.out‚134.01904296875 disk.dm-0.out‚ 21096.859375 disk.md0p1.out‚ 128.12890625 disk.sda.out‚134.01904296875$ memory_percent‚91.98737144470215$ disk./.usageGB‚2713.284076690674" disk.nvme0n1.out‚ 13446.8203125 disk.md0.in‚0! disk.sdb2.out‚134.01904296875 disk.sda.in‚0 network.sent‚ 1.699419e+06 disk.rbd2.out‚ 0.57080078125 disk.md0.out‚ 128.12890625! disk.nvme1n1.out‚ 7650.0390625 disk.rbd0.out‚ 0.62109375( proc.memory.availableMB‚ 1312.7890625) proc.memory.percent‚18.17641258239746 disk.sdb1.in‚0 disk.md0p1.in‚0<Ðqj“Š‚(:ˆ ÊûêÔϽõÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀN"ï‹‚(:‹ ÊûêÔ»õàà disk.sdb1.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚13675.81640625 disk.sdb2.in‚0 disk.sdb2.out‚135.1064453125% disk./.usageGB‚2713.3555450439453 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.md0.in‚0 disk.sdb.in‚0) proc.memory.availableMB‚ 1311.78515625 disk.rbd0.out‚ 0.62109375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚21326.01953125 disk.sda.in‚0) proc.memory.percent‚18.17803382873535 network.sent‚ 1.725006e+06 network.recv‚540194 disk.rbd0.in‚0 disk.rbd2.out‚ 0.57080078125" proc.memory.rssMB‚ 2978.2890625 proc.cpu.threads‚17 disk.sdb.out‚135.1064453125$ memory_percent‚91.99349880218506 disk.md0p1.out‚ 129.1484375 disk.sda2.out‚135.1064453125 disk.dm-0.in‚0) disk./.usagePercent‚75.86655666694881 disk.md0p1.in‚0 disk.md0.out‚ 129.1484375 disk.sda.out‚135.1064453125 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 7650.203125 cpu‚25.061520703912887 disk.sdb1.in‚0©‹mTŒ‚(:… ÙûêÔŠõ”ß gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡§@‚(:Œ ÙûêÔ¾ï…á disk.rbd2.in‚0 disk.rbd2.out‚ 0.57080078125 disk.dm-0.out‚21589.79296875 disk.nvme1n1.in‚0 disk.md0.in‚0 network.recv‚544745 disk.rbd0.out‚ 0.62109375 disk.sda2.out‚ 136.263671875 disk.sdb.out‚ 136.263671875 disk.sda.in‚0 disk.sda.out‚ 136.263671875 disk.md0p1.out‚ 130.26171875# disk.nvme0n1.out‚13939.41015625 cpu‚25.104698318677933 network.sent‚ 1.733189e+06! disk.nvme1n1.out‚ 7650.3828125 disk.sdb1.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1382.17578125 proc.cpu.threads‚17 disk.sdb2.out‚ 136.263671875 disk.sda1.out‚0 disk.md0.out‚ 130.26171875$ memory_percent‚91.56386852264404 disk.sdb.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.sdb2.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2908.14453125* proc.memory.percent‚17.749905586242676) disk./.usagePercent‚75.87005619900773% disk./.usageGB‚2713.4807052612305 disk.md0p1.in‚0 disk.sdb1.out‚0Í=ûÒ”Ž‚(:‰ èûêÔ…ÌØà gpu.0.gpu‚57.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰-K ‚(:‚ èûêÔÁŠ³á disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0* proc.memory.percent‚18.041682243347168 network.recv‚549061$ disk./.usageGB‚2713.856456756592 disk.dm-0.in‚0 disk.md0p1.out‚ 131.62890625 disk.sda2.out‚ 137.67578125 proc.cpu.threads‚17" disk.nvme1n1.out‚ 7650.54296875 disk.sda2.in‚0 disk.md0.in‚0 disk.sda.in‚0 disk.sda.out‚ 137.67578125 disk.nvme1n1.in‚0 disk.sdb.out‚ 137.67578125" disk.nvme0n1.out‚ 14251.5703125 disk.sda1.in‚0 disk.md0.out‚ 131.62890625# proc.memory.rssMB‚ 2955.94921875 cpu‚25.14647509853398$ memory_percent‚91.85371398925781 network.sent‚ 1.737174e+06 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.57080078125% proc.memory.availableMB‚ 1334.6875) disk./.usagePercent‚75.88056236808225 disk.dm-0.out‚21902.11328125 disk.sdb.in‚0 disk.sdb2.out‚ 137.67578125 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.62109375½‡§pbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPpr1¿u‘bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP`ħ^k’bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JP²ƒ¸`u“bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPUÙá‡e”b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JP4ÌÅDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP‡d7sºbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP>‰œŠq»bl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPjIk¼bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JP!zL±t½bo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPh>Í¢l¾bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JPlêK®l¿bg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JPán8kÀbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP97·ŠnÁbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPêñ —yÂbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPŸÅJ pÃbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JP'vPkÄbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPŽ²0 hÅbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JP´Î¼_zÆbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPöíÿÌdÇb_ š8yÑbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPgü9ŽoÒbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPrX¸4lÓbg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPc€™+oÔbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPöÛ,0vÕbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPË·Š´kÖbf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JPßÐ&q×bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JP’]sØbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPc­­oÙbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JP!‚(nÚbi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPötºpÛbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP+#2bxÜbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPi,ÅcÝb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JP" üþxÞbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP¢V†­wßbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP²²1åtàbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPt‚¶rvábq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPoi±hâbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPr†Žsãbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÕþþþdäb_ lÿbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP(¯di€bd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPÞ/½rbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JP¥ƒjÉm‚bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JP‘¥Æ*nƒbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPû ]§n„bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JPûY x…bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPý¢SÓf†ba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPpÇyFt‡bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPH< —qˆbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP^ü”y‰bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP}h€ÑqŠbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP71Ùƒ2‹- _wandbruntime‚1451  _runtime‚1451pÅ(ŒÈT YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 trainer/global_step‚1208 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚25162716 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚102822907600896 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚99480028166384 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚98104020207440 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 700639744 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189001216 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚101441735092736 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30572890 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 180164096 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚98104003168080 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 _step‚96 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55715681 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚101441735092736 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1381873147904 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55735608 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚111951202955776 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚102823608240640 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1381873147904 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 682636800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30552965 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 700639744 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚113399677221888 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1381190511104 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448630490112 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698103532 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 _runtime‚1451.977513627 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25162718 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22458185 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚7383949 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508ï‰úìR*ÌT _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 _runtime‚1451.977513627 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25162718 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22458185 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚7383949 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 trainer/global_step‚1208 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚25162716 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚102822907600896 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚99480028166384 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚98104020207440 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 700639744 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189001216 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚101441735092736 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30572890 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 180164096 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚98104003168080 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 _step‚96 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55715681 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚101441735092736 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1381873147904 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55735608 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚111951202955776 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚102823608240640 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1381873147904 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 682636800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30552965 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 700639744 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚113399677221888 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1381190511104 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448630490112 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698103532`Øo„ŠiŽbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP´•¹Ûpbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP6ôÏÍrbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPŒ3¹ur‘bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPMEDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPŽW#w br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPûóò‡j¡be BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPíõºki¢bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPƒÇ›r£bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPŠ¼(7k¤bf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPJj³•m¥bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP³xTn¦bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JP%:Ãíw§br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPQ³Ÿw¨br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¾*‘q©bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPÒŠŸ$sªbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP Û>Kh«bc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPòV0/t¬bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP)( àj­be BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP)sq&k®bf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÝX]àr¯bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPI&Rs°bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPó&/|±bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP„r¶Pz²bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP…šÐ·b³b] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP£Zýp´bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPÀYàDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPLýgÀj„be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPys]ãu…bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP•äI¨v†bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP ÒI³q‡bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JP »Ô‘oˆbj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPE˜3ä2‰- _wandbruntime‚1453  _runtime‚1453ˆm0ª¶ŠÒR SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25178194 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚30561779 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚112007494449664 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚97 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187720845 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25178192 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14134971392 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚25178192 VHDeviceStatsMonitÛšu¢#or.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1381680361984 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚216 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚102878358998528 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 682636800 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15083800 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15083798 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚7389342 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚112007470509568 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚102877658358784 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1382362998784 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 682636800 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698103532 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚98154446312624 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚101495995999744 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1376513209188 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14134971392 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚101495995999744 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684408973 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚101495977996800 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449120340992 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 156224000 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30751 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1381680361984 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55739971 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1382362998784 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚102877658358784 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚30561779 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚102878358998528 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 700639744 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚98154429273264 trainer/global_step‚1208 _runtime‚1453.118169498 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30581704 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22473140 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚7389556 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚113456614790656 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚101495977996800 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189001216 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚214 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1448964116992 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚30581704 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22473356 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 180164096 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0­yÀ\)‹ÖR _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚101495977996800 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449120340992 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 156224000 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30751 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1381680361984 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55739971 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1382362998784 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚102877658358784 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚30561779 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚102878358998528 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 700639744 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚98154429273264 trainer/global_step‚1208 _runtime‚1453.118169498 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30581704 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22473140 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚7389556 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚113456614790656 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚101495977996800 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189001216 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚214 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1448964116992 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚30581704 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22473356 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 180164096 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25178194 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚30561779 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚112007494449664 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚97 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187720845 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25178192 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14134971392 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚25178192 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1381680361984 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚216 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚102878358998528 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 682636800 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15083800 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15083798 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚7389342 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚112007470509568 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚102877658358784 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1382362998784 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 682636800 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698103532 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚98154446312624 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚101495995999744 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1376513209188 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14134971392 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚101495995999744 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684408973 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712aXn\’Œ‚(:‡ ÷ûêÔ¨Ü¢ß gpu.0.gpu‚61.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îâe¶‚(:Œ ÷ûêÔ–áÜß# proc.memory.rssMB‚ 2969.75390625 disk.sda.out‚138.83154296875 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.sdb2.in‚0 disk.rbd0.out‚ 0.62109375$ memory_percent‚91.94085597991943 network.recv‚552683! disk.sda2.out‚138.83154296875! disk.sdb2.out‚138.83154296875 disk.sda1.in‚0 disk.rbd2.out‚ 0.57080078125# disk./.usageGB‚2713.85640335083 disk.dm-0.in‚0 disk.md0p1.out‚ 132.71484375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1320.41015625 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.sdb.in‚0 disk.sdb.out‚138.83154296875" disk.nvme1n1.out‚ 7650.94921875 cpu‚25.19102746294827( disk./.usagePercent‚75.8805608748348# disk.nvme0n1.out‚14668.73828125 disk.md0.out‚ 132.71484375 disk.nvme1n1.in‚0) proc.memory.percent‚18.12593936920166 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.out‚ 22319.6875 disk.rbd2.in‚0 network.sent‚ 1.742478e+06 disk.sda.in‚0'ýk¦NŽbI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP–fìˆ;b6 jet/accuracy_1_of_4"trainer/global_step2:JPÝË?@b; particle/accuracy_3_of_5"trainer/global_step2:JPŽ–5ñH‘bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¿8=;’b6 jet/accuracy_1_of_7"trainer/global_step2:JPßÞH“bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÙ0u…H”bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPšìÉ N•bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÏpí J–bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP)ŠzH—bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¿ãÊÑJ˜bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÿØ¿mN™bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP›(D•NšbI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÓ1{]H›bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP^ *GJœbE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÕí53HbC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP,vBJžbE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÈ8NNŸbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP}ÔOÿJ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÛ,ìN¡bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPœPðÓN¢bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP[û¡ÖN£bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPSk×N¤bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP}n´@¥b; particle/accuracy_4_of_6"trainer/global_step2:JPžR1hH¦bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP$ZXtH§bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP’Ѐ*J¨bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÖÙº=J©bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¸Ê­NªbI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP…æí;«b6 jet/accuracy_6_of_6"trainer/global_step2:JP<¦ÓnJ¬bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP‡܃N­bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÃ@ƒÜH®bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP6ùJ¯bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPË'ÐÕH°bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÚ HdJ±bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPúòèH²bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPΟՈJ³bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP×\²%J´bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‡¤ž¢NµbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP^ôYñH¶bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPlQ´=H·bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPE³^J¸bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPMp&N¹bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÜjŸNºbI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP;ª ìH»bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPé\$àH¼bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPméJ½bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP1I[í;¾b6 jet/accuracy_1_of_1"trainer/global_step2:JPMËH¿bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPSÍ^HÀbC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPóeßAHÁbC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPDêR JÂbE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPO& HÃbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPq¨ý'JÄbE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¡ã„JÅbE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP }ÝøJÆbE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¦1Ï-JÇbE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPŸåJÈbE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPôU¸‰;Éb6 jet/accuracy_3_of_3"trainer/global_step2:JPDmìJHÊbC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPƒ i5JËbE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP?WðNÌbI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPwK‚`HÍbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPŠ\i5<Îb7 particle/sensitivity"trainer/global_step2:JPÁlŸTHÏbC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPƒäKëNÐbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPù<TNÑbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPV‚FŠNÒbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP0oüLJÓbE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPŽ¹QHÔbC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP„UJÕbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPv->œHÖbC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP•¦$N×bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPZ=}æNØbI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP£ðNÙbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÃGkdNÚbI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÏÔãÑHÛbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇ|¥hJÜbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPž:wõHÝbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¥‡HÞbC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPÍGfÿJßbE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPq´fJàbE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPˆÊ+—HábC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP·1À¸NâbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPþQhc@ãb; particle/accuracy_1_of_5"trainer/global_step2:JPú¬ ¼HäbC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÊ;ËJåbE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP@§¯öHæbC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPȬàNçbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP€6h¡JèbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPLû·*JébE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÔÊ€NêbI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP~›ËHëbC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÊÚcJìbE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¢ŠÞHíbC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÑCÇNJîbE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP^‡ÚhHïbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPd´vJðbE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPõ<Õ^HñbC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPòè×’JòbE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÀgg;JóbE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP \JôbE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP)²‹ÍHõbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP8¨ÙTHöbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP–é¨N÷bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP[$@øb; particle/accuracy_5_of_5"trainer/global_step2:JP.o³HùbC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPúËÃäHúbC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP” ¢{JûbE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPŸCVNübI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPXÐWJýbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÛ‚èáJþbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPåkFVHÿbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‘éH€ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP ‰éjJ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP „ŸJ‚ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¨#ÈëHƒ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP›Ò¤J„ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŒ#ô¸J… bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPœs¯¢H† bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÙ tJ‡ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPpI@Nˆ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP]Vô¾N‰ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPg“HŠ bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPh2´ú;‹ b6 jet/accuracy_2_of_6"trainer/global_step2:JP®S9tHŒ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPåÙí%J bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPä}dHŽ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPˆ÷N bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPjêpJ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÌÙyJ‘ bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¿ÊN’ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPšþƒCN“ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPj+6ÐJ” bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPäLð<• b7 particle/specificity"trainer/global_step2:JP—ýH– bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¿/j€N— bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¨±?J˜ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP”chQ;™ b6 jet/accuracy_2_of_4"trainer/global_step2:JP×CôÃHš bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP8B³@› b; particle/accuracy_2_of_6"trainer/global_step2:JPÑŸº­Hœ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÌÐáH bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPB¾&z;ž bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_sõz6ktep2:JPK|Z˜NŸ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPw‡¿›TJÄ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPgbmNÅ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP[[=.@Æ b; particle/accuracy_1_of_4"trainer/global_step2:JP‚ÛÇäJÇ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP½‘ÓÝJÈ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPàǬ;É b6 jet/accuracy_6_of_7"trainer/global_step2:JPêìÆHÊ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP~jHË bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPA~>àJÌ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPe9åèNÍ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÞQĨJÎ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPH×¾«NÏ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP…ûšNÐ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPn:ÏÑNÑ bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÅøëJÒ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPQè{KNÓ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPt¡ÆZNÔ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPeJÕ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPnn:NÖ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÝVe%CJØ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰e kJÙ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPg@NÚ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP’HÛ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÉ–gCJÜ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPà£ÀDNÝ bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP|ùÉNÞ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÕçNß bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP3Òž$Nà bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP†ƒHá bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPäf“Nâ bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPIX£¨Hã bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPPœÐ@ä b; particle/accuracy_1_of_6"trainer/global_step2:JPÅâSRHå bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP`P÷Hæ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP^øÍUJç bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP=Fj¨;è b6 jet/accuracy_1_of_3"trainer/global_step2:JPµíTHé bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÉØ>ÌHê bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP…ooQJë bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP([Jì bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPI&%°Ní bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP…x`Nî bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¶aKHï bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPX…ÛÇHð bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP¸¤Jñ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPp/pHò bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÓÐ#Hó bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP–~¡Jô bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPh³Z¾Jõ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPbñ£ÈJö bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP©®Â¯N÷ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPš60WJø bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP'¤¢Nù bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP‡Øl]Nú bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP1[$áHû bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPä< ©Jü bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPŸÎj;ý b6 jet/accuracy_3_of_7"trainer/global_step2:JP5ß„@þ b; particle/accuracy_6_of_7"trainer/global_step2:JP×E·Jÿ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPsóŽ“J€ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPLºÄâH bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPq¿Ú×J‚ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7ª‰´Nƒ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP½9PÂH„ bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP»  J… bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP² ÉJ† bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP‘oˆH‡ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP…#^LJˆ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP”‚zOJ‰ bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP1ýRKJŠ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP þÖ„N‹ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP‹“hòNŒ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPuÒEæN bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPJŽ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÙ†]aH bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP;O¥íJ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP'2²J‘ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP¯ì¡GJ’ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP5FxH“ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP=‡sH” bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPë]þJ• bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP $ƒH– bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÍ.iJ— bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPhAx¦H˜ bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPüѤ™J™ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÈ+íNš bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP°Âb ÂÂHÝ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP?ˆ«SJÞ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÈKƒuHß bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPcÙU;à b6 jet/accuracy_2_of_2"trainer/global_step2:JPnÁ¤qHá bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP»Ý¯ÅHâ bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPïœHã bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÓmwÑJä bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPw¬¢Nå bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPûi‹9æ b4 particle/accuracy"trainer/global_step2:JPˆ®À¥Gç bB validation_average_jet_accuracy"trainer/global_step2:JPò£¼|Jè bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPí<‘\Né bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPΛ³Hê bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP[˜ŽdNë bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPæ4ãáJì bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‡SD}Ní bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP ž¿}@¯ b; particle/accuracy_2_of_4"trainer/global_step2:JP¬·ƒJ° bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¯¿%;H± bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPëÝ­¸H² bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPJÊs±N³ bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPY>§¿<´ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_stl¤3‚ep2:JPˆ,Ê}Nµ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPò˜¥?N¶ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÚ€J"J· bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPcnXEH¸ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPIö\H¹ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPìD’¢Hº bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPþœ³íN» bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP®BùÊN¼ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP-øbJJ½ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPOú!3H¾ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP[RоN¿ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP1‹þãNÀ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPëeNÁ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPQ`~`N bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÿ„);à b6 jet/accuracy_1_of_5"trainer/global_step2:JP,’5HÄ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPE‡Dl;Å b6 loss/val_total_loss"trainer/global_step2:JPáÇÚJÆ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPîçüSHÇ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP„Œ,iJÈ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPªV&hHÉ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPWø¶JÊ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP·»ƒfHË bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPù‡…JÌ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÞžÛNÍ bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPýX˜ãHÎ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPoøxJÏ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPž„NÐ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPd—?ÂNÑ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP_±®NÒ bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP}’gc@Ó b; particle/accuracy_5_of_7"trainer/global_step2:JPŒ—ÓÊHÔ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP–- ©HÕ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPØc?HÖ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕºäåN× bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPPçûNØ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPG¸Å¾NÙ bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP?95—NÚ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPºÉ°JÛ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJÜ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPoÇ®–JÝ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¶MäªHÞ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPôê…Nß bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPàÎU @à b; particle/accuracy_2_of_2"trainer/global_step2:JP•ÃÒž;á b6 jet/accuracy_4_of_7"trainer/global_step2:JP',øHâ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP§³|Hã bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPW¾ßNä bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÁ¯yÁJå bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP>OHæ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPØÑ×Nç bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPsßÃNè bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP¡ &@Jé bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÒÎü>Jê bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ë¦öHë bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP«žHÌ;ì b6 jet/accuracy_7_of_7"trainer/global_step2:JPk.ñHí bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPjÈF}Hî bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP…'µNï bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¨©÷cHð bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP×±$öJñ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP+F|Jò bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¡)0ªHó bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPîg¯Hô bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP°2ƒNõ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPjŒ`ÞNö bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPR¡7?N÷ bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¯‘ Ç;ø b6 jet/accuracy_3_of_4"trainer/global_step2:JP. œ@ù b; particle/accuracy_2_of_3"trainer/global_step2:JP¥ _,Nú bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP(ê÷¼Nû bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPmáË›Jü bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPVÕÅHý bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP{¹ÂôNþ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPû¡ñ“Jÿ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP2où­N€ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP!‰åóJ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP,ŠM0N‚ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP§åNƒ bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPjœÍN„ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPmOZvJ… bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP÷+Š¹N† bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP8v5ON‡ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPiY±Hˆ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÒm°ïN‰ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPŽ¿èAJŠ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPá×uÆJ‹ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPèÌÁHŒ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿ìü H bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPdwOûNŽ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÿ9\;H bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP² jèJ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPä[±J‘ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPD=ÖH’ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP\ÓkJ“ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPy-¡¯N” bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP†Æîò@• b; particle/accuracy_2_of_7"trainer/global_step2:JP‘;ÿ/H– bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¼¨íJ— bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPj$p~N˜ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPñD¡æH™ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPSmRóJš bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÕïêVH› bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPòElJœ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPnÑx³N bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPâ¨ÒpNž bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPœÔ—£NŸ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÜAöóH  bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPà¿l¨N¡ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÛ!=J¢ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¢!é&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2923525273799896 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4142959713935852 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1995144933462143 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 +jet/accuracy_2_of_5‚0.35327693819999695 jet/accuracy_5_of_6‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08327478915452957 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17677022516727448 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14259257912635803 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5515109896659851 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_7‚1 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18185028433799744 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18478159606456757 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3708333373069763 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02313748374581337 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12228123098611832 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31979167461395264 *jet/accuracy_1_of_2‚0.6779447793960571 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_7‚1 $ _timestamp‚1.788526085359723e+09 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977195262908936 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4172247648239136 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954551458358765 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20488503575325012 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3714223802089691 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1756756752729416 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.35087719559669495 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.401845782995224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3558686375617981 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3819582462310791 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3422619104385376 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.3125 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.22631579637527466 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013951877132058144 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1916147768497467 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05877285078167915 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777777642011642 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23907321691513062 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30304229259490967 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1846504807472229 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.3341005742549896 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666716337204 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33574575185775757 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0909644290804863 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17624405026435852 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.33908936381340027 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4265971779823303 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10057472437620163 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009014178067445755 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.097087062895298 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4117640554904938 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12394318729639053 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1644754707813263 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1245793029665947 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977195262908936 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024144761264324188 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3775297701358795 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40740740299224854 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886053800582886 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4194004237651825 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12187092006206512 +jet/accuracy_2_of_2‚0.07972612977027893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9880661368370056 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 _step‚98 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05634898692369461 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 'particle/accuracy‚0.825433075428009 6validation_average_jet_accuracy‚0.3749426007270813 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1746043711900711 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3471478223800659 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9874160289764404 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3673401474952698 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3985498249530792 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2839147448539734 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28557151556015015 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3099702000617981 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981366991996765 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3065055310726166 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12423143535852432 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27387383580207825 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603134393692 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871125221252441 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40962857007980347 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11378607153892517 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2224138081073761 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07663771510124207 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008974310010671616 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08760108053684235 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_3_of_6‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3847857117652893 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17722442746162415 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39080458879470825 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17884233593940735 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1520923376083374 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3419330418109894 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36385229229927063 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927180409431458 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8990271091461182 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *loss/val_total_loss‚55.262132654964525 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.30995607376098633 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4592505991458893 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714663565159 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09773663431406021 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_5_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1793426126241684 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1530342996120453 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.35007262229919434 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3673401474952698 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005043117795139551 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13740384578704834 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 jet/accuracy_7_of_7‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31375330686569214 =&Purity/2FRt1SRqqt0FBt/e“h©µHvent_proportion‚0.0318080373108387 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3020833730697632 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_3_of_4‚0.04551537707448006 particle/accuracy_2_of_3‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14737653732299805 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3394850790500641 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.30666667222976685 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907673597335815 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38594120740890503 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543211936950684 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.042692940682172775 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20204676687717438 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17550504207611084 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3777998089790344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9849297404289246 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9816879034042358 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04007504880428314 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 -jet/accuracy_4_of_4‚0.0009920635493472219 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14278952777385712 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24575164914131165 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.40615418553352356 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943179488182068 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12447617202997208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4975806772708893 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9854512214660645 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.990770161151886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.05019841343164444 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9230769276618958 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09361705929040909 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8332270383834839 particle/accuracy_3_of_5‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 jet/accuracy_1_of_7‚1 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32332518696784973 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30029788613319397 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010138248093426228 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983993649482727 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.30406901240348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2982769310474396 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29906198382377625 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3332659602165222 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870210289955139 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42240312695503235 *jet/accuracy_1_of_1‚0.4079741835594177 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.01718885824084282 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18381638824939728 +particle/sensitivity‚0.2808246910572052 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2370370775461197 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.34022992849349976 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.32575759291648865 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.24814815819263458 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33145058155059814 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3909241557121277 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022763067856431007 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 epoch‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10248085856437683 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.060283686965703964 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9942434430122375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14968477189540863 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_5_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.029520666226744652 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21069121360778809 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928266406059265 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2580459713935852 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.22455409169197083 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26550328731536865 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9962533712387085 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.3589743971824646 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05977541580796242 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40386760234832764 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.363008975982666 +particle/specificity‚0.9261822700500488 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4032224714756012 *jet/accuracy_2_of_4‚0.3155844509601593 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2906549870967865 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6045138835906982 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16044971346855164 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37996602058410645 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.37345680594444275 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3119569420814514 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3603418171405792 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.24393942952156067 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8122494220733643 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13473796844482422 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0909644290804863 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6287878751754761 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5237519145011902 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6140860319137573 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4553447663784027 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 trainer/global_step‚1208 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3126327097415924 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17105786502361298 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012107424437999725 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16666993498802185 particle/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894604682922363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2927631437778473 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3334403932094574 )jet/accuracy_1_of_3‚0.778863787651062 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34062638878822327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4295156002044678 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20047537982463837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15567561984062195 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2944551110267639 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3145407438278198 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004995408467948437 jet/accuracy_3_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4001370072364807 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9762225151062012Ë–èuc7ß ¹¦ 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.3589743971824646 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05977541580796242 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40386760234832764 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.363008975982666 +particle/specificity‚0.9261822700500488 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4032224714756012 *jet/accuracy_2_of_4‚0.3155844509601593 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2906549870967865 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6045138835906982 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16044971346855164 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37996602058410645 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.37345680594444275 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3119569420814514 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3603418171405792 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.24393942952156067 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8122494220733643 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13473796844482422 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.0909644290804863 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6287878751754761 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5237519145011902 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6140860319137573 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4553447663784027 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 trainer/global_step‚1208 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3126327097415924 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17105786502361298 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012107424437999725 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16666993498802185 particle/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894604682922363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2927631437778473 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3334403932094574 )jet/accuracy_1_of_3‚0.778863787651062 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34062638878822327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4295156002044678 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20047537982463837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15567561984062195 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2944551110267639 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3145407438278198 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004995408467948437 jet/accuracy_3_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4001370072364807 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9762225151062012 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.23428989946842194 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12702707946300507 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962379932403564 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6653786301612854 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 _runtime‚1468.898381378 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3357890248298645 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.29949596524238586 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0017636684933677316 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001976859290152788 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2923525273799896 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4142959713935852 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1995144933462143 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 +jet/accuracy_2_of_5‚0.35327693819999695 jet/accuracy_5_of_6‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08327478915452957 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17677022516727448 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14259257912635803 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5515109896659851 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_7‚1 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18185028433799744 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18478159606456757 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3708333373069763 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02313748374581337 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12228123098611832 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31979167461395264 *jet/accuracy_1_of_2‚0.6779447793960571 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_7‚1 $ _timestamp‚1.788526085359723e+09 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977195262908936 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4172247648239136 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954551458358765 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20488503575325012 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3714223802089691 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1756756752729416 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.35087719559669495 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.401845782995224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3558686375617981 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3819582462310791 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3422619104385376 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.3125 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.22631579637527466 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013951877132058144 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1916147768497467 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2¨~ÀÓÝ[SRqqt2FBt/FBt_purity‚0.9900000095367432 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05877285078167915 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777777642011642 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23907321691513062 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30304229259490967 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1846504807472229 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.3341005742549896 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666716337204 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33574575185775757 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.0909644290804863 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17624405026435852 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.33908936381340027 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4265971779823303 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10057472437620163 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009014178067445755 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.097087062895298 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4117640554904938 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12394318729639053 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1644754707813263 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1245793029665947 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977195262908936 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024144761264324188 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3775297701358795 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40740740299224854 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886053800582886 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4194004237651825 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12187092006206512 +jet/accuracy_2_of_2‚0.07972612977027893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9880661368370056 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 _step‚98 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05634898692369461 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 'particle/accuracy‚0.825433075428009 6validation_average_jet_accuracy‚0.3749426007270813 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1746043711900711 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3471478223800659 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9874160289764404 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3673401474952698 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3985498249530792 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2839147448539734 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28557151556015015 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3099702000617981 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981366991996765 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3065055310726166 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12423143535852432 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27387383580207825 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603134393692 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871125221252441 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40962857007980347 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11378607153892517 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2224138081073761 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07663771510124207 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008974310010671616 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08760108053684235 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_3_of_6‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3847857117652893 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17722442746162415 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39080458879470825 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17884233593940735 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1520923376083374 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3419330418109894 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36385229229927063 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927180409431458 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8990271091461182 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *loss/val_total_loss‚55.262132654964525 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.30995607376098633 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4592505991458893 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714663565159 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09773663431406021 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_5_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1793426126241684 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1530342996120453 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.35007262229919434 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3673401474952698 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005043117795139551 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13740384578704834 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 jet/accuracy_7_of_7‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31375330686569214 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3020833730697632 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_3_of_4‚0.04551537707448006 particle/accuracy_2_of_3‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14737653732299805 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3394850790500641 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.30666667222976685 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907673597335815 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38594120740890503 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543211936950684 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.042692940682172775 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20204676687717438 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17550504207611084 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3777998089790344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9849297404289246 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9816879034042358 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04007504880428314 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 -jet/accuracy_4_of_4‚0.0009920635493472219 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14278952777385712 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24575164914131165 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.40615418553352356 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943179488182068 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12447617202997208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4975806772708893 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9854512214660645 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.990770161151886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.05019841343164444 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9230769276618958 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09361705929040909 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8332270383834839 particle/accuracy_3_of_5‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 jet/accuracy_1_of_7‚1 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32332518696784973 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30029788613319397 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010138248093426228 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983993649482727 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.30406901240348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2982769310474396 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29906198382377625 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3332659602165222 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870210289955139 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42240312695503235 *jet/accuracy_1_of_1‚0.4079741835594177 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.01718885824084282 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18381638824939728 +particle/sensitivity‚0.2808246910572052 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2370370775461197 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.34022992849349976 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.32575759291648865 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.24814815819263458 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33145058155059814 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3909241557121277 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022763067856431007 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 epoch‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10248085856437683 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.060283686965703964 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9942434430122375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14968477189540863 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_5_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.029520666226744652 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21069121360778809 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9928266406059265 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2580459713935852 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.22455409169197083 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26550328731536865 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9962533712387085b‹ZÄ_à ‚(: †üêÔ€„ûß disk.rbd2.in‚0' proc.memory.availableMB‚ 1234.640625 disk.rbd0.in‚0 disk.nvme1n1.in‚0 network.sent‚ 1.812116e+06) disk./.usagePercent‚75.86582049595611$ disk./.usageGB‚2713.329216003418 disk.sdb2.in‚0 disk.sda.in‚0 disk.md0p1.in‚0 disk.sdb.out‚140.00732421875 disk.rbd0.out‚ 0.62109375 disk.sda1.out‚0 disk.rbd2.out‚ 0.57080078125$ memory_percent‚92.46435165405273" proc.memory.rssMB‚ 3053.1796875# disk.nvme0n1.out‚14748.83203125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚22400.02734375 disk.md0.out‚ 133.84765625* proc.memory.percent‚18.635129928588867 network.recv‚558540 disk.md0.in‚0! disk.sda2.out‚140.00732421875 disk.sdb.in‚0! disk.sdb2.out‚140.00732421875 disk.sdb1.in‚0 disk.nvme0n1.in‚0 cpu‚25.22859345390071 disk.dm-0.in‚0 disk.sda.out‚140.00732421875! disk.nvme1n1.out‚ 7651.1953125 disk.md0p1.out‚ 133.84765625 disk.sdb1.out‚0 disk.sda1.in‚0>Ë«ÈŽá ‚(:ƒ †üêÔÕ„“à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚34.0 gpu.0.powerWatts‚85.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eJ$ ”â ‚(:‰ •üêÔô´üÞ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.327' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.83175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¹¨Ýã ‚(: •üêÔÈÖ¯à disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 network.sent‚ 1.935491e+06 disk.sdb2.in‚0" disk.nvme1n1.out‚ 7651.45703125 disk.rbd2.in‚0" disk.nvme0n1.out‚ 15812.5390625 network.recv‚565320 disk.dm-0.out‚23461.24609375 disk.sdb.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2980.328125 disk.nvme0n1.in‚0 disk.sda.out‚141.72705078125) disk./.usagePercent‚75.90165128849335 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚91.91839694976807* proc.memory.percent‚18.190479278564453 disk.sdb.out‚141.72705078125 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 0.57080078125! disk.sda2.out‚141.72705078125 proc.cpu.threads‚17$ disk./.usageGB‚2714.610698699951 disk.rbd0.out‚ 0.62109375 disk.sda1.out‚0) proc.memory.availableMB‚ 1324.08984375! disk.sdb2.out‚141.72705078125 disk.md0p1.out‚ 135.50390625 cpu‚25.22448425032508 disk.md0.in‚0 disk.md0.out‚ 135.50390625øÖµ”ä ‚(:‰ ¤üêÔýåÃà gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F®s<å ‚(:‡ ¤üêÔÖ¥á disk.rbd0.in‚0 disk.md0.out‚ 136.60546875 disk.sdb.in‚0 cpu‚25.222571063764043 disk.rbd0.out‚ 0.62109375 disk.sda2.in‚0 disk.sdb.out‚142.9013671875 disk.sda.in‚0 disk.sda.out‚142.9013671875 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 16508.5234375 disk.sda2.out‚142.9013671875( proc.memory.availableMB‚ 1323.4609375 network.recv‚569761$ disk./.usageGB‚2714.636089324951! proc.memory.rssMB‚ 2980.328125* proc.memory.percent‚18.190479278564453 disk.md0p1.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.sdb2.out‚142.9013671875 disk.rbd2.out‚ 0.57080078125 disk.dm-0.in‚0 disk.dm-0.out‚ 24224.2421875 proc.cpu.threads‚17) disk./.usagePercent‚75.90236122099516 disk.nvme1n1.out‚ 7715.71875 disk.sdb1.in‚0 disk.nvme0n1.in‚0# memory_percent‚91.9222354888916 network.sent‚ 1.943678e+06 disk.sdb2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 136.60546875ƒÏ¹i“æ ‚(:ˆ ³üêÔÔÅòà gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_°šÆç ‚(:‘ ³üêÔÓȶá disk.sdb.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 16771.1640625 disk.sdb2.in‚0* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 network.recv‚573821! disk.sda2.out‚144.14990234375 disk.sdb.out‚144.14990234375 disk.rbd2.out‚ 0.57080078125 disk.dm-0.out‚24549.05859375! disk.sdb2.out‚144.14990234375 disk.sda.out‚144.14990234375 disk.nvme1n1.in‚0 disk.md0p1.in‚0# proc.memory.rssMB‚ 2980.33203125 disk.dm-0.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.92280769348145 cpu‚25.220722078679838% disk./.usageGB‚2714.7634086608887 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 1323.3671875" disk.nvme1n1.out‚ 7777.89453125 disk.md0p1.out‚ 137.7890625 disk.rbd0.in‚0 network.sent‚ 1.949023e+06) disk./.usagePercent‚75.90592112291526 disk.sda2.in‚0 disk.md0.out‚ 137.7890625¾$óQ2è - _wandbruntime‚1521  _runtime‚1521éú›Æé À & lr-AdamW/pg1‚0.0007317750738329083 & lr-AdamW/pg2‚0.0007317750738329083 trainer/global_step‚1249 % _timestamp‚1.7885261376807287e+09 _runtime‚1521.216279781 _step‚99œ0°oÊê Ä & lr-AdamW/pg1‚0.0007317750738329083 & lr-AdamW/pg2‚0.0007317750738329083 trainer/global_step‚1249 % _timestamp‚1.7885261376807287e+09 _runtime‚1521.216279781 _step‚99cÊSj2ë - _wandbruntime‚1521  _runtime‚1521jÝû²! ì æO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31695607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26185782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885261376815066e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 trainer/global_step‚1249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106899757519872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428246634573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1432696325632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 eQDeviceStatåøóÜËsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116348000826880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106899757519872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101970672422416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105466151223808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7690403 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101970689461776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31669265 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105466133220864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106898829546496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105466151223808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1427338659577 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432696325632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117850339296768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944ØS ð'í êO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106899757519872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101970672422416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105466151223808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7690403 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101970689461776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31669265 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105466133220864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106898829546496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105466151223808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1427338659577 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432696325632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117850339296768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31695607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26185782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885261376815066e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 trainer/global_step‚1249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106899757519872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428246634573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1432696325632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116348000826880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570d&Öaò2î - _wandbruntime‚1522  _runtime‚1522áYHÌö&ï ðM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚1522.172042487 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1503492646912 % _timestamp‚1.7885261386358325e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15702771 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26205940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26205940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31694096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433817785856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102049707056320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31720464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433817785856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1503251294208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1434727769600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57900036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1429363908059 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15702769 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7696121 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117941453885440~Ø»8úð ôM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15702771 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26205940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26205940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31694096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433817785856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102049707056320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31720464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433817785856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1503251294208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1434727769600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57900036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1429363908059 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15702769 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7696121 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117941453885440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚1522.172042487 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1503492646912 % _timestamp‚1.7885261386358325e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144e¾¦SÑ2ñ - _wandbruntime‚1522  _runtime‚1522n =lDò ¾ /loss/FRt2/assignment_loss‚5.966064298475111 2loss/SRqqt4/assignment_loss‚1.9341792054474354 /loss/FBt2/detection_loss‚2.0031923562288285 _runtime‚1522.173600754 /loss/FRt1/detection_loss‚3.2849562813838324 0loss/SRqqt1/detection_loss‚3.326817954841413 %loss/total_loss‚51.39609995691079 /loss/FRt1/assignment_loss‚4.123473887642224 /loss/FRt4/assignment_loss‚3.267543177996109 .loss/FRt3/detection_loss‚2.040964792554195 /loss/FBt1/detection_loss‚3.9604738454023996 /loss/FRt3/assignment_loss‚2.464583769440651 2loss/SRqqt1/assignment_loss‚1.6667044868594723 1loss/SRqqt3/assignment_loss‚1.176674806567676 .loss/FRt2/detection_loss‚2.652029588415816 1loss/SRqqt3/detection_loss‚0.9857350531965494 trainer/global_step‚1249 _step‚102 0loss/FBt2/assignment_loss‚0.7514843807948637 loss/FBt4/assignment_loss‚0 epoch‚1 0loss/SRqqt2/detection_loss‚2.532246986031532 2loss/SRqqt2/assignment_loss‚2.2887935519218443 0loss/FBt1/assignment_loss‚0.5861326425025849 0loss/FBt3/assignment_loss‚0.3322407083499797 /loss/FRt4/detection_loss‚1.7487726090336912 .loss/FBt3/detection_loss‚0.762388091778168 /loss/FBt4/detection_loss‚1.9706088304519653 1loss/SRqqt4/detection_loss‚1.5700386515944391 % _timestamp‚1.7885261386383219e+097=ÅvHó  /loss/FRt3/assignment_loss‚2.464583769440651 2loss/SRqqt1/assignment_loss‚1.6667044868594723 1loss/SRqqt3/assignment_loss‚1.176674806567676 .loss/FRt2/detection_loss‚2.652029588415816 1loss/SRqqt3/detection_loss‚0.9857350531965494 trainer/global_step‚1249 _step‚102 0loss/FBt2/assignment_loss‚0.7514843807948637 loss/FBt4/assignment_loss‚0 epoch‚1 0loss/SRqqt2/detection_loss‚2.532246986031532 2loss/SRqqt2/assignment_loss‚2.2887935519218443 0loss/FBt1/assignment_loss‚0.5861326425025849 0loss/FBt3/assignment_loss‚0.3322407083499797 /loss/FRt4/detection_loss‚1.7487726090336912 .loss/FBt3/detection_loss‚0.762388091778168 /loss/FBt4/detection_loss‚1.9706088304519653 1loss/SRqqt4/detection_loss‚1.5700386515944391 % _timestamp‚1.7885261386383219e+09 /loss/FRt2/assignment_loss‚5.966064298475111 2loss/SRqqt4/assignment_loss‚1.9341792054474354 /loss/FBt2/detection_loss‚2.0031923562288285 _runtime‚1522.173600754 /loss/FRt1/detection_loss‚3.2849562813838324 0loss/SRqqt1/detection_loss‚3.326817954841413 %loss/total_loss‚51.39609995691079 /loss/FRt1/assignment_loss‚4.123473887642224 /loss/FRt4/assignment_loss‚3.267543177996109 .loss/FRt3/detection_loss‚2.040964792554195 /loss/FBt1/detection_loss‚3.9604738454023996f€Ò€K“ô ‚(:ˆ ÂüêÔ’¤½à gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.386' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒM”4 õ ‚(: ÂüêÔƃóà disk.md0.in‚0 disk.md0p1.out‚ 138.9296875$ memory_percent‚91.92492961883545% disk./.usageGB‚2713.8340187072754 disk.sdb2.in‚0 disk.sda.in‚0 network.sent‚ 1.965334e+06 disk.sda2.out‚145.3447265625 disk.sdb.in‚0 disk.rbd0.in‚0 disk.sdb.out‚145.3447265625 disk.sda1.out‚0 disk.rbd2.out‚ 0.63330078125) disk./.usagePercent‚75.87993499083231 disk.sda.out‚145.3447265625 disk.nvme1n1.in‚0 disk.nvme1n1.out‚7933 disk.rbd0.out‚ 0.62109375 disk.sda1.in‚0! disk.nvme0n1.out‚ 16859.390625 cpu‚25.21877008180744 disk.sda2.in‚0 disk.md0.out‚ 138.9296875 disk.dm-0.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 network.recv‚577794 disk.dm-0.out‚ 24792.390625 disk.sdb2.out‚145.3447265625 disk.md0p1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1323.01953125 disk.sdb1.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2980.33203125u6 “ö ‚(:ˆ ÑüêÔô¡ãà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–lˆf÷ ‚(: ÑüêÔד‘â disk.md0p1.in‚0 disk.rbd2.in‚0 cpu‚25.216955584271112! disk.sda2.out‚146.81103515625 disk.sdb1.out‚0# proc.memory.rssMB‚ 2980.33203125$ disk./.usageGB‚2714.085006713867 disk.sdb2.in‚0) proc.memory.availableMB‚ 1323.66015625 disk.dm-0.out‚ 24792.9375 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.88695272054326 disk.nvme0n1.in‚0 disk.dm-0.in‚0" disk.nvme1n1.out‚ 7933.40234375 disk.rbd2.out‚ 0.63330078125 network.sent‚ 2.061455e+06 disk.sdb.in‚0 disk.sdb.out‚146.81103515625! disk.sdb2.out‚146.81103515625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 140.32421875$ memory_percent‚91.92101955413818* proc.memory.percent‚18.190503120422363 network.recv‚584404 disk.md0.in‚0 disk.sda.in‚0# disk.nvme0n1.out‚16859.53515625 disk.sda2.in‚0 disk.sda.out‚146.81103515625 disk.rbd0.out‚ 0.62109375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 140.32421875CÆ””ø ‚(:‰ àüêÔ— éà gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô›Óù ‚(:Š àüêÔ±§£á) disk./.usagePercent‚75.88438988127744 disk.sdb.in‚0 disk.sda.in‚0! disk.nvme1n1.out‚ 8140.2109375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.out‚ 141.60546875 proc.cpu.threads‚17 network.recv‚588942 disk.sda2.out‚ 148.154296875 disk.rbd0.in‚0% disk./.usageGB‚2713.9933471679688# disk.nvme0n1.out‚16860.55078125 disk.md0.in‚0 disk.md0p1.out‚ 141.60546875 disk.rbd2.out‚ 0.6572265625 disk.sda2.in‚0 disk.sdb2.out‚ 148.154296875 disk.sdb1.out‚0 network.sent‚ 2.067532e+06 disk.nvme0n1.in‚0 disk.dm-0.out‚25000.76171875 disk.md0p1.in‚0$ memory_percent‚91.92166328430176 disk.dm-0.in‚0 disk.sdb.out‚ 148.154296875 disk.sda.out‚ 148.154296875 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.62109375* proc.memory.percent‚18.190503120422363 cpu‚25.215087533677252 disk.sdb2.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1323.5546875# proc.memory.rssMB‚ 2980.33203125dÄØ2ú - _wandbruntime‚1573  _runtime‚1573×@vÇû Á _runtime‚1573.697494549 _step‚103 & lr-AdamW/pg1‚0.0007316759499470062 & lr-AdamW/pg2‚0.0007316759499470062 trainer/global_step‚1299 % _timestamp‚1.7885261901619241e+09øíÏËü Å _step‚103 & lr-AdamW/pg1‚0.0007316759499470062 & lr-AdamW/pg2‚0.0007316759499470062 trainer/global_step‚1299 % _timestamp‚1.7885261901619241e+09 _runtime‚1573.697494549gŸsÂ2ý - _wandbruntime‚1573  _runtime‚1573÷ï³#ì'þ æO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27193782 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109547087512064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32914167 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120859262109184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1560428647936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚1573.698779674 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16293956 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111035871262720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27193780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109547105515008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109547105515008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1489693721088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32914167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1560195670528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1488783750656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7975303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120859286049280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107405693078657 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105921569188976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111036799236096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60107947 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885261901631315e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24268631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16293954 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32940509 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27193780Ü´ÍWú&ÿ êO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16293956 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111035871262720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27193780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109547105515008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109547105515008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1489693721088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32914167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1560195670528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1488783750656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7975303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120859286049280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107405693078657 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105921569188976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111036799236096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60107947 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885261901631315e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24268631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16293954 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32940509 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27193780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27193782 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109547087512064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32914167 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120859262109184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1560428647936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚1573.698779674 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570h¸îø2€- _wandbruntime‚1574  _runtime‚1574`e\+ö&ðM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60179308 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107484902932123 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1485241163167 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885261911633363e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109628706597888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111119539795456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106000586783520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1490815194624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27213942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32938998 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107485827946687 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27213942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32965366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚1299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111119539795456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106000569744160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1561358345728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1484333187963 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109628724600832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27213940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60152938 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120949487813632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111118611808768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _runtime‚1574.699825646 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107485827946687 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27213942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32965366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚1299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111119539795456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106000569744160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1561358345728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1484333187963 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109628724600832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27213940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60152938 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120949487813632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111118611808768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _runtime‚1574.699825646 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60179308 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107484902932123 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1485241163167 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885261911633363e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247iÌ™‰²2ƒ- _wandbruntime‚1574  _runtime‚1574÷‘.ÉG„Á 1loss/SRqqt1/detection_loss‚2.6049164216965437 loss/FBt4/assignment_loss‚0 1loss/FBt1/assignment_loss‚0.17664667769920264 _runtime‚1574.700933791 _step‚106 2loss/SRqqt3/assignment_loss‚1.4186530839356164 2loss/SRqqt4/assignment_loss‚1.7700772100083886 -loss/FRt1/detection_loss‚2.98444429397583 /loss/FRt2/detection_loss‚2.8868945980894156 /loss/FRt3/detection_loss‚1.6594075873832812 &loss/total_loss‚48.273437386302874 0loss/FRt3/assignment_loss‚1.3296987230127508 1loss/SRqqt1/assignment_loss‚2.095842274837196 0loss/FBt2/assignment_loss‚0.4111934210278605 /loss/FRt4/detection_loss‚2.0608200538903474 1loss/SRqqt4/detection_loss‚1.2700218611529894 % _timestamp‚1.7885261911656435e+09 1loss/SRqqt2/assignment_loss‚2.316176862716675 1loss/SRqqt2/detection_loss‚3.0814078283309936 /loss/FBt2/detection_loss‚2.6912496127188206 /loss/FBt4/detection_loss‚1.7000192403793335 /loss/FRt1/assignment_loss‚4.050031008720398 -loss/FRt4/assignment_loss‚3.3520758608977 1loss/FBt3/assignment_loss‚0.21479871289072242 /loss/FBt1/detection_loss‚3.0464772527868096 epoch‚1 trainer/global_step‚1299 1loss/SRqqt3/detection_loss‚1.1784114139893698 .loss/FRt2/assignment_loss‚5.02444066261423 /loss/FBt3/detection_loss‚0.9497327235484019-ÌHK…Å /loss/FRt1/assignment_loss‚4.050031008720398 -loss/FRt4/assignment_loss‚3.3520758608977 1loss/FBt3/assignment_loss‚0.21479871289072242 /loss/FBt1/detection_loss‚3.0464772527868096 epoch‚1 trainer/global_step‚1299 1loss/SRqqt3/detection_loss‚1.1784114139893698 .loss/FRt2/assignment_loss‚5.02444066261423 /loss/FBt3/detection_loss‚0.9497327235484019 1loss/SRqqt1/detection_loss‚2.6049164216965437 loss/FBt4/assignment_loss‚0 1loss/FBt1/assignment_loss‚0.17664667769920264 _runtime‚1574.700933791 _step‚106 2loss/SRqqt3/assignment_loss‚1.4186530839356164 2loss/SRqqt4/assignment_loss‚1.7700772100083886 -loss/FRt1/detection_loss‚2.98444429397583 /loss/FRt2/detection_loss‚2.8868945980894156 /loss/FRt3/detection_loss‚1.6594075873832812 &loss/total_loss‚48.273437386302874 0loss/FRt3/assignment_loss‚1.3296987230127508 1loss/SRqqt1/assignment_loss‚2.095842274837196 0loss/FBt2/assignment_loss‚0.4111934210278605 /loss/FRt4/detection_loss‚2.0608200538903474 1loss/SRqqt4/detection_loss‚1.2700218611529894 % _timestamp‚1.7885261911656435e+09 1loss/SRqqt2/assignment_loss‚2.316176862716675 1loss/SRqqt2/detection_loss‚3.0814078283309936 /loss/FBt2/detection_loss‚2.6912496127188206 /loss/FBt4/detection_loss‚1.7000192403793335j¼yü/”†‚(:‰ ïüêÔ«„¨á gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÿÁ͇‚(:† ïüêÔ¡›€â disk.sda1.in‚0 network.recv‚593088) disk./.usagePercent‚75.87357951636531 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.nvme0n1.out‚ 16932.9453125 cpu‚25.213358484751208 network.sent‚ 2.083815e+06" disk.nvme1n1.out‚ 8184.19921875 disk.md0p1.out‚ 142.8515625 disk.rbd0.out‚ 0.62109375 disk.rbd2.out‚ 0.6572265625$ disk./.usageGB‚2713.606716156006 disk.md0.in‚0 disk.nvme1n1.in‚0 disk.md0p1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 disk.sdb.in‚0 disk.sdb2.out‚ 149.458984375 disk.sda1.out‚0 disk.sda2.out‚ 149.458984375' proc.memory.availableMB‚ 1323.453125 disk.md0.out‚ 142.8515625 disk.sdb.out‚ 149.458984375 disk.sdb2.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2980.33203125 disk.dm-0.out‚25117.14453125 disk.nvme0n1.in‚0$ memory_percent‚91.92228317260742 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 149.458984375 disk.dm-0.in‚0NyK¦’ˆ‚(:‡ þüêÔ§í¢à gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ¯å‰‚(:‡ þüêÔͦÚà disk.md0p1.out‚ 144.453125 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚151.1103515625 disk.sdb2.in‚0 cpu‚25.211572829873315 disk.sdb.out‚151.1103515625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.md0p1.in‚0 disk.md0.out‚ 144.453125 disk.dm-0.out‚25189.83984375 disk.sdb2.out‚151.1103515625$ memory_percent‚91.92118644714355# proc.memory.rssMB‚ 2980.33203125 disk.nvme1n1.in‚0# disk.nvme0n1.out‚17004.90234375 disk.sda2.in‚0( proc.memory.availableMB‚ 1323.6328125* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17) disk./.usagePercent‚75.87232358859957 disk.rbd2.out‚ 0.72021484375 disk.sdb.in‚0 disk.sdb1.in‚0$ disk./.usageGB‚2713.561798095703 disk.sda.out‚151.1103515625 disk.rbd2.in‚0 network.sent‚ 2.180179e+06 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 network.recv‚599515 disk.sda.in‚0 disk.nvme1n1.out‚ 8184.9375&c”Š‚(:‰ ýêÔ²ü³à gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾vž‹‚(:… ýêÔüŒÔá network.sent‚ 2.184183e+06% disk./.usageGB‚2713.6382446289062 disk.sdb1.in‚0* proc.memory.percent‚18.190503120422363( disk./.usagePercent‚75.8744610656633 disk.sdb1.out‚0 disk.sda2.in‚0 disk.md0.out‚ 145.921875 disk.dm-0.in‚0 disk.dm-0.out‚ 25266.5703125# memory_percent‚91.9234037399292 cpu‚25.209901108856364 disk.md0p1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.md0p1.out‚ 145.921875 disk.rbd0.out‚ 0.62109375 disk.rbd2.in‚0 disk.sda2.out‚152.6494140625 disk.sdb2.in‚0# proc.memory.rssMB‚ 2980.33203125 disk.sdb.in‚0 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 8256.9375 disk.sda1.out‚0 disk.rbd2.out‚ 0.72021484375" disk.nvme0n1.out‚ 17009.6328125 disk.md0.in‚0 network.recv‚604118) proc.memory.availableMB‚ 1323.26953125 disk.sdb.out‚152.6494140625 disk.sdb2.out‚152.6494140625 disk.sda.out‚152.6494140625FP(”Œ‚(:‰ œýêÔÖéäÞ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šl’ü‚(: œýêÔ¾©Ëà! disk.sda2.out‚153.89697265625) disk./.usagePercent‚75.88847924607899 disk.sda.out‚153.89697265625 disk.md0p1.out‚ 147.1171875 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚153.89697265625 network.sent‚ 2.192386e+06$ disk./.usageGB‚2714.139602661133 disk.rbd0.in‚0 disk.sda1.in‚0 disk.md0.in‚0 disk.md0.out‚ 147.1171875 disk.sdb.out‚153.89697265625 disk.rbd0.out‚ 0.62109375 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚25372.95703125 disk.sdb2.in‚0* proc.memory.percent‚18.190503120422363 network.recv‚608344 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 2980.33203125 cpu‚25.208323988417597 disk.sda.in‚0" disk.nvme1n1.out‚ 8310.91015625 disk.nvme0n1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1324.3046875! disk.nvme0n1.out‚ 17062.046875$ memory_percent‚91.91708564758301 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.md0p1.in‚0G12Ž- _wandbruntime‚1627  _runtime‚1627ÈÙ¬ÆÀ $ _timestamp‚1.788526243467365e+09 _runtime‚1627.002960789 _step‚107 & lr-AdamW/pg1‚0.0007315005988466614 & lr-AdamW/pg2‚0.0007315005988466614 trainer/global_step‚13498ÂFîÊÄ trainer/global_step‚1349 $ _timestamp‚1.788526243467365e+09 _runtime‚1627.002960789 _step‚107 & lr-AdamW/pg1‚0.0007315005988466614 & lr-AdamW/pg2‚0.0007315005988466614kßa=2‘- _wandbruntime‚1627  _runtime‚1627¯íýX ’æO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1540001144789 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1627.004173385 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126989218190848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1618646919168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113628041803264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28201782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113628041803264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125370547331584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34159069 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8259468 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109872431876816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25156674 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111411525046609 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1539093169793 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28201780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126988959176192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34159069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1544871175680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115173840952320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1545781146112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1618411844608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1545781146112 trainer/global_step‚1349 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115172912978944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28201782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 % _timestamp‚1.7885262434685447e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125370571271680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109872448916176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25156068 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8258864>ùyÿð'“êO trainer/global_step‚1349 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115172912978944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28201782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 % _timestamp‚1.7885262434685447e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125370571271680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109872448916176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25156068 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8258864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1540001144789 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1627.004173385 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126989218190848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1618646919168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113628041803264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28201782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113628041803264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125370547331584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34159069 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8259468 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109872431876816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25156674 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111411525046609 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1539093169793 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28201780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126988959176192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34159069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1544871175680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115173840952320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1545781146112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1618411844608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1545781146112lCø–Í2”- _wandbruntime‚1627  _runtime‚1627Ï*L=ö&•ðM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125460773036032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 trainer/global_step‚1349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1546902619648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111491659914431 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28221942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1546902619648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174442 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264563 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8265171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113709660889088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115256581511680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16909269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1619588745216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28221940 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115255653524992 % _timestamp‚1.7885262444566207e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109951466510720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34210268 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125460796976128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111492584928995 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 _runtime‚1627.993068751 _step‚109 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1545992635904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0lŽ†Üä–ôM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109951466510720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34210268 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125460796976128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111492584928995 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 _runtime‚1627.993068751 _step‚109 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1545992635904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125460773036032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 trainer/global_step‚1349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1546902619648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111491659914431 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28221942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 YGDeviceS’“utatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1546902619648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174442 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264563 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8265171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113709660889088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115256581511680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16909269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1619588745216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28221940 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115255653524992 % _timestamp‚1.7885262444566207e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0m7‡2—- _wandbruntime‚1627  _runtime‚1627snÙY˜Ó 2loss/FBt4/assignment_loss‚0.016363548097918585 2loss/SRqqt3/assignment_loss‚1.4546515420079231 0loss/FRt1/assignment_loss‚4.8259579965046475 /loss/FRt4/detection_loss‚1.8683559765962705 trainer/global_step‚1349 0loss/FRt3/assignment_loss‚2.0323672511360864 0loss/SRqqt1/detection_loss‚2.420625422682081 1loss/SRqqt3/detection_loss‚0.9417665080382274 epoch‚1 % _timestamp‚1.7885262444592855e+09 _step‚110 0loss/SRqqt2/detection_loss‚2.693300123872428 1loss/SRqqt4/detection_loss‚1.4755773858492633 2loss/SRqqt4/assignment_loss‚1.4719499998770833 .loss/FRt3/detection_loss‚1.910449679195881 0loss/FBt2/assignment_loss‚0.8660331283263726 2loss/SRqqt2/assignment_loss‚2.2053682721894363 0loss/FBt1/assignment_loss‚0.4671118378715359 /loss/FBt1/detection_loss‚2.9820063497338976 .loss/FBt4/detection_loss‚4.925426423549652 %loss/total_loss‚53.30781514142427 _runtime‚1627.994647589 0loss/FBt3/assignment_loss‚0.2859881599142227 /loss/FBt2/detection_loss‚2.3626688378197804 /loss/FBt3/detection_loss‚0.8953151555869746 /loss/FRt1/detection_loss‚3.1302035876682828 /loss/FRt2/detection_loss‚2.5713887129511153 .loss/FRt2/assignment_loss‚5.50556492805481 /loss/FRt4/assignment_loss‚3.326953887122951 1loss/SRqqt1/assignment_loss‚2.672420426777431yhwt]™× 2loss/SRqqt2/assignment_loss‚2.2053682721894363 0loss/FBt1/assignment_loss‚0.4671118378715359 /loss/FBt1/detection_loss‚2.9820063497338976 .loss/FBt4/detection_loss‚4.925426423549652 %loss/total_loss‚53.30781514142427 _runtime‚1627.994647589 0loss/FBt3/assignment_loss‚0.2859881599142227 /loss/FBt2/detection_loss‚2.3626688378197804 /loss/FBt3/detection_loss‚0.8953151555869746 /loss/FRt1/detection_loss‚3.1302035876682828 /loss/FRt2/detection_loss‚2.5713887129511153 .loss/FRt2/assignment_loss‚5.50556492805481 /loss/FRt4/assignment_loss‚3.326953887122951 1loss/SRqqt1/assignment_loss‚2.672420426777431 2loss/FBt4/assignment_loss‚0.016363548097918585 2loss/SRqqt3/assignment_loss‚1.4546515420079231 0loss/FRt1/assignment_loss‚4.8259579965046475 /loss/FRt4/detection_loss‚1.8683559765962705 trainer/global_step‚1349 0loss/FRt3/assignment_loss‚2.0323672511360864 0loss/SRqqt1/detection_loss‚2.420625422682081 1loss/SRqqt3/detection_loss‚0.9417665080382274 epoch‚1 % _timestamp‚1.7885262444592855e+09 _step‚110 0loss/SRqqt2/detection_loss‚2.693300123872428 1loss/SRqqt4/detection_loss‚1.4755773858492633 2loss/SRqqt4/assignment_loss‚1.4719499998770833 .loss/FRt3/detection_loss‚1.910449679195881 0loss/FBt2/assignment_loss‚0.8660331283263726n¥Á\’š‚(:‡ «ýêÔÒŸ×ß gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.552' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ\iö›‚(:Œ «ýêÔë­à disk.rbd2.in‚0$ memory_percent‚91.91741943359375 disk.rbd0.in‚0* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 disk.md0.out‚ 148.69921875 disk.dm-0.out‚25712.45703125 disk.sda.in‚0! disk.nvme1n1.out‚ 8440.5078125 disk.sda1.out‚0 disk.nvme0n1.in‚0% disk./.usageGB‚2714.6390953063965! disk.sda2.out‚155.54638671875 disk.dm-0.in‚0 cpu‚25.206605745024067 network.sent‚ 2.195572e+06 disk.sda2.in‚0 disk.sdb.in‚0 disk.md0p1.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.rbd2.out‚ 0.783203125 network.recv‚612370) disk./.usagePercent‚75.90244526949448# proc.memory.rssMB‚ 2980.33203125# disk.nvme0n1.out‚17271.94921875# proc.memory.availableMB‚1324.25 disk.sda1.in‚0 disk.md0.in‚0 disk.sdb2.in‚0! disk.sdb2.out‚155.54638671875 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sdb.out‚155.54638671875 disk.sda.out‚155.54638671875 disk.md0p1.out‚ 148.69921875Cœ‚(:‰ ºýêÔîÅüÞ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:0z&‚(:… ºýêÔž›õà disk.md0p1.out‚ 149.9140625 disk.md0.in‚0* proc.memory.percent‚18.190503120422363 disk.dm-0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚ 156.798828125 disk.sdb.in‚0 disk.sdb.out‚ 156.798828125 disk.sdb2.out‚ 156.798828125 disk.nvme1n1.in‚0 disk.sda1.out‚0! disk.nvme1n1.out‚ 8543.9140625 disk.rbd0.out‚ 0.62109375 disk.sda2.in‚0# proc.memory.rssMB‚ 2980.33203125 cpu‚25.205028517592478 network.sent‚ 2.304863e+06% disk./.usageGB‚2714.6381340026855 disk.rbd0.in‚0 disk.sda2.out‚ 156.798828125) disk./.usagePercent‚75.90241839104038 disk.dm-0.out‚ 25975.390625 disk.sda.in‚0 disk.md0p1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0$ memory_percent‚91.91815853118896 proc.cpu.threads‚17 disk.sdb2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125" disk.nvme0n1.out‚ 17433.6015625 disk.md0.out‚ 149.9140625) proc.memory.availableMB‚ 1324.12890625 network.recv‚618614U¼‹Óž‚(:… ÉýêÔ뺒à gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.14 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôeÄ_Ÿ‚(:ˆ ÉýêÔÞœà! disk.sdb2.out‚158.03857421875 disk.sda.out‚158.03857421875! disk.sda2.out‚158.03857421875 disk.sda.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.out‚158.03857421875# proc.memory.rssMB‚ 2980.33203125" disk.nvme1n1.out‚ 8725.09765625 disk.sda1.out‚0 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.out‚ 17586.8125 disk.nvme0n1.in‚0* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.md0.out‚ 151.09375) disk./.usagePercent‚75.90165715482262 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 151.09375 disk.sdb.in‚0$ memory_percent‚91.91591739654541) proc.memory.availableMB‚ 1324.49609375 network.sent‚ 2.308867e+06 network.recv‚623116$ disk./.usageGB‚2714.610908508301 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.md0.in‚0 cpu‚25.203376874603357 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚26311.91015625 9Àá2 - _wandbruntime‚1679  _runtime‚1679³ Ç¡Á & lr-AdamW/pg2‚0.0007312490570747806 trainer/global_step‚1399 % _timestamp‚1.7885262957729747e+09 _runtime‚1679.308405512 _step‚111 & lr-AdamW/pg1‚0.0007312490570747806P,ÓéˢŠ& lr-AdamW/pg1‚0.0007312490570747806 & lr-AdamW/pg2‚0.0007312490570747806 trainer/global_step‚1399 % _timestamp‚1.7885262957729747e+09 _runtime‚1679.308405512 _step‚111o~NY«2£- _wandbruntime‚1679  _runtime‚1679çB7Më'¤åO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129881856494080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117709014097408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239268864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117709014097408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601868571136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1594970424901 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17500456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29209782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26046102 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131558558385152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129881832553984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26045489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1679.309432157 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119309954695168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29209780 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1600958600704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788526295773684e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115418282028917 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29209780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117708996094464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17500454 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8545035 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113823328643376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1595878399897 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117708996094464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35403971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601868571136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1676725831168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35430313 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115419207043273 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344Æ”‹ìï'¥éO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117709014097408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239268864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117709014097408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601868571136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1594970424901 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17500456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29209782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26046102 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131558558385152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129881832553984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26045489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1679.309432157 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119309954695168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29209780 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1600958600704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788526295773684e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115418282028917 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29209780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117708996094464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17500454 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8545035 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113823328643376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1595878399897 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117708996094464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35403971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601868571136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1676725831168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35430313 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115419207043273 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129881856494080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570p”ĦÑ˦‚(:‰ ØýêÔ”Óàß gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.ä0*‰É047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0< ü;§‚(:Ž ØýêÔ–’à disk.sda1.out‚0 disk.sda2.in‚0! disk.sda2.out‚159.45556640625 cpu‚25.201736606903918 network.recv‚627466# disk.nvme0n1.out‚17634.84765625) disk./.usagePercent‚75.90040016045158% disk./.usageGB‚2714.5659523010254 disk.md0p1.out‚ 152.453125 disk.rbd2.in‚0 disk.sdb.in‚0 disk.sda.out‚159.45556640625 disk.md0p1.in‚0 disk.md0.in‚0* proc.memory.percent‚18.190503120422363! disk.sdb2.out‚159.45556640625 disk.nvme1n1.in‚0 disk.md0.out‚ 152.453125 disk.sdb2.in‚0! disk.nvme1n1.out‚ 8769.3203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0) proc.memory.availableMB‚ 1324.37890625# proc.memory.rssMB‚ 2980.33203125 disk.dm-0.in‚0$ memory_percent‚91.91663265228271 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.62109375 disk.sda1.in‚0 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚26404.16796875 disk.sdb.out‚159.45556640625 proc.cpu.threads‚17 network.sent‚ 2.315063e+06¥’Ê 2¨- _wandbruntime‚1680  _runtime‚1680ÆvÛ?ö&©ðM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885262967459257e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1678130168320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117790633183232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119393623227904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29229940 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1602990044672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35428802 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64658742 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26063266 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35455170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17512519 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649968045568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1602080060928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119393623227904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1602990044672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129972058258432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1677909787136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29229940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚113 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129972082198528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29229942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115498416896739 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35428802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115499341911303 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119392695241216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117790615180288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640¾–5ú&ªôM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚113 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129972082198528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29229942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115498416896739 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35428802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115499341911303 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119392695241216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117790615180288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885262967459257e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1678130168320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117790633183232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119393623227904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29229940 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1602990044672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35428802 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64658742 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26063266 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35455170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17512519 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649968045568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1602080060928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119393623227904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1602990044672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129972058258432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1677909787136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29229940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600qÑåS@2«- _wandbruntime‚1680  _runtime‚1680ý~¾T¬Î 2loss/SRqqt4/assignment_loss‚1.7232232602934043 0loss/FBt2/assignment_loss‚0.6240175316731129 0loss/SRqqt3/detection_loss‚1.178680264376677 0loss/SRqqt4/detection_loss‚1.514921427204989 # _timestamp‚1.78852629674852e+09 0loss/FBt1/assignment_loss‚0.4759491529355942 .loss/FRt2/detection_loss‚3.110606692054055 &loss/total_loss‚52.097050860729524 trainer/global_step‚1399 /loss/FRt1/assignment_loss‚4.205834823269998 2loss/SRqqt2/assignment_loss‚1.8264944950739543 2loss/SRqqt3/assignment_loss‚1.1548471078276634 0loss/FBt3/assignment_loss‚0.3402715476291059 /loss/FRt3/detection_loss‚1.9946037494473987 /loss/FBt2/detection_loss‚2.3800568928321204 epoch‚1 _step‚114 .loss/FRt1/detection_loss‚2.988512962095199 0loss/FRt3/assignment_loss‚1.5321565137969122 1loss/SRqqt1/detection_loss‚2.4132690644926496 /loss/FRt2/assignment_loss‚5.126451270146803 0loss/FRt4/assignment_loss‚3.4068169647997077 /loss/FRt4/detection_loss‚1.9250547675923868 1loss/SRqqt2/detection_loss‚2.6202940474718046 .loss/FBt4/detection_loss‚4.682643711566925 _runtime‚1680.283840701 0loss/SRqqt1/assignment_loss‚2.11553788681825 /loss/FBt3/detection_loss‚0.7730578174350494 0loss/FBt4/assignment_loss‚1.6073953269806225 .loss/FBt1/detection_loss‚2.376353582915138Ÿ|â°X­Ò /loss/FBt3/detection_loss‚0.7730578174350494 0loss/FBt4/assignment_loss‚1.6073953269806225 .loss/FBt1/detection_loss‚2.376353582915138 2loss/SRqqt4/assignment_loss‚1.7232232602934043 0loss/FBt2/assignment_loss‚0.6240175316731129 0loss/SRqqt3/detection_loss‚1.178680264376677 0loss/SRqqt4/detection_loss‚1.514921427204989 # _timestamp‚1.78852629674852e+09 0loss/FBt1/assignment_loss‚0.4759491529355942 .loss/FRt2/detection_loss‚3.110606692054055 &loss/total_loss‚52.097050860729524 trainer/global_step‚1399 /loss/FRt1/assignment_loss‚4.205834823269998 2loss/SRqqt2/assignment_loss‚1.8264944950739543 2loss/SRqqt3/assignment_loss‚1.1548471078276634 0loss/FBt3/assignment_loss‚0.3402715476291059 /loss/FRt3/detection_loss‚1.9946037494473987 /loss/FBt2/detection_loss‚2.3800568928321204 epoch‚1 _step‚114 .loss/FRt1/detection_loss‚2.988512962095199 0loss/FRt3/assignment_loss‚1.5321565137969122 1loss/SRqqt1/detection_loss‚2.4132690644926496 /loss/FRt2/assignment_loss‚5.126451270146803 0loss/FRt4/assignment_loss‚3.4068169647997077 /loss/FRt4/detection_loss‚1.9250547675923868 1loss/SRqqt2/detection_loss‚2.6202940474718046 .loss/FBt4/detection_loss‚4.682643711566925 _runtime‚1680.283840701 0loss/SRqqt1/assignment_loss‚2.11553788681825rX馦‘®‚(:† çýêÔëÓôà gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É¡¤ÿ¯‚(:Œ çýêÔ­¹áâ disk.sdb2.in‚0 disk.nvme1n1.in‚0 network.recv‚633569 disk.dm-0.out‚ 26638.140625 disk.rbd0.in‚0# disk.nvme0n1.out‚17796.60546875 disk.md0.in‚0 disk.sdb2.out‚161.5341796875 disk.sda.out‚161.5341796875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 1324.1953125 proc.cpu.threads‚17 disk.sda2.in‚0% disk./.usageGB‚2714.6361541748047 disk.sdb.in‚0 disk.rbd0.out‚ 0.62109375$ memory_percent‚91.91775321960449 disk.sda.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0* proc.memory.percent‚18.190503120422363 network.sent‚ 2.410714e+06 disk.sda2.out‚161.5341796875 disk.md0p1.out‚ 154.46484375 disk.md0.out‚ 154.46484375# proc.memory.rssMB‚ 2980.33203125 disk.sda1.in‚0 disk.rbd2.out‚ 0.783203125" disk.nvme1n1.out‚ 8841.53515625) disk./.usagePercent‚75.90236303422421 disk.dm-0.in‚0 disk.sdb.out‚161.5341796875 disk.md0p1.in‚0 cpu‚25.200332024474278Oöè‘”°‚(:‰ öýêÔáùÕà gpu.0.gpu‚21.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùW\q±‚(:õ öýêÔ¥¶¿á disk.rbd2.out‚ 0.783203125" disk.nvme0n1.out‚ 18058.6640625 disk.md0p1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2714.6377449035645 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.sdb2.in‚0! proc.memory.availableMB‚8.625 disk.nvme1n1.out‚8843.875 network.recv‚639261 disk.sdb1.in‚0 disk.sda2.out‚ 162.798828125 disk.dm-0.in‚0 disk.dm-0.out‚ 26902.5390625 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2980.33203125 network.sent‚ 2.509423e+06( disk./.usagePercent‚75.9024075116661 disk.sdb.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.out‚ 162.798828125 disk.sda.in‚0 disk.sda.out‚ 162.798828125* proc.memory.percent‚18.190503120422363 disk.rbd0.out‚ 0.62109375 disk.md0p1.out‚ 155.671875$ memory_percent‚99.94735717773438 disk.md0.out‚ 155.671875 disk.sdb.out‚ 162.798828125 cpu‚25.1951765589474 disk.md0.in‚0ú܇”²‚(:‰ …þêÔî··à gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÛA³‚(:Œ …þêÔŸ¨ñá# disk.nvme0n1.out‚18162.95703125 disk.md0.in‚0 disk.dm-0.in‚0 disk.sdb.out‚163.99853515625 disk.rbd0.out‚ 0.62109375 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0 disk.sdb1.out‚0 cpu‚25.19377722565238 network.recv‚644150 disk.nvme0n1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2980.33203125 network.sent‚ 2.515497e+06 disk.rbd2.in‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚ 156.81640625# memory_percent‚99.9474287033081& proc.memory.availableMB‚ 8.61328125! disk.sda2.out‚163.99853515625 disk.md0.out‚ 156.81640625! disk.sdb2.out‚163.99853515625 disk.sda.out‚163.99853515625 disk.sda1.in‚0% disk./.usageGB‚2714.7629737854004 disk.dm-0.out‚27058.94140625 disk.sdb.in‚0 disk.nvme1n1.out‚ 8895.984375) disk./.usagePercent‚75.90590896361459 disk.sda.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.190503120422363 disk.md0p1.in‚0 proc.cpu.threads‚17ñÓ¥2´- _wandbruntime‚1736  _runtime‚1736D!+õǵÁ trainer/global_step‚1449 % _timestamp‚1.7885263528234112e+09 _runtime‚1736.359041011 _step‚115 & lr-AdamW/pg1‚0.0007309213770522917 & lr-AdamW/pg2‚0.0007309213770522917XK^ê˶Š_runtime‚1736.359041011 _step‚115 & lr-AdamW/pg1‚0.0007309213770522917 & lr-AdamW/pg2‚0.0007309213770522917 trainer/global_step‚1449 % _timestamp‚1.7885263528234112e+09st†Jï2·- _wandbruntime‚1736  _runtime‚17362ƒÏ—¼¸æO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 _step‚116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121789950385664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚1736.360483707 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1657955996160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26936968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121789950385664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1657955996160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30217780 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30217782 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18103706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18103704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1657046025728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8832649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36648873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123446996411392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30217782 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26936353 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44®‘îð'¹êO _runtime‚1736.360483707 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1657955996160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26936968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121789950385664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1657955996160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30217780 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30217782 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18103706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18103704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1657046025728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8832649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36648873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123446996411392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30217782 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26936353 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 _step‚116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121789950385664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0t&yfƒ2º- _wandbruntime‚1737  _runtime‚1737‚¯2ö&»ðM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18115771 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30237940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36700072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136219619423744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1652872928491 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1658167485952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123529736957440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1651964953287 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117853225965120 trainer/global_step‚1449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121871587474432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121871569471488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8838957 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30237942 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚1737.412527911 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26954728åcHC•¼ôM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚1737.412527911 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26954728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18115771 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30237940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36700072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136219619423744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1652872928491 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small0;Kªe_pool.freed‚ 1658167485952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123529736957440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1651964953287 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117853225965120 trainer/global_step‚1449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121871587474432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121871569471488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8838957 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30237942 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9121406 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138904426938880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127584966100992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793843948544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37893775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140698024455680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885264082473912e+09܈qCá$ÍêO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121725088097776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125870904676864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246431744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31225782 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31225780 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127584966100992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31225782 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125870922679808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27828964 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 _runtime‚1791.783192992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9121406 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138904426938880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127584966100992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793843948544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37893775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GĘ×åADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140698024455680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885264082473912e+09xIqÎ2Î- _wandbruntime‚1792  _runtime‚1792ãB°ö&ÏðM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127667706660352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121804088652960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125952523762688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37918606 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37944974 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31245942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125952541765632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37944974 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37918606 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666778673664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 _runtime‚1792.763100355 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1714254910976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18719019 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1499 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140789667983360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138994652643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1715164894720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9127144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9127752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121804105692320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31245940 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125952523762688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885264092268581e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127667706660352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27846773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1714254910976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1794784473088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31245940 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1707842208395 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123511930861355 *‹Œú&ÐôM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125952523762688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37918606 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37944974 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31245942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125952541765632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37944974 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37918606 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666778673664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 _runtime‚1792.763100355 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1714254910976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18719019 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1499 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140789667983360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138994652643328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1715164894720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9127144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9127752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121804105692320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31245940 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125952523762688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885264092268581e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127667706660352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27846773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1714254910976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1794784473088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31245940 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1707842208395 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123511930861355 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127667706660352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121804088652960yÑbCí2Ñ- _wandbruntime‚1792  _runtime‚1792zL¿TÒÎ 2loss/SRqqt1/assignment_loss‚2.5838875885193167 2loss/SRqqt3/assignment_loss‚1.3383138136123307 2loss/SRqqt4/assignment_loss‚1.6393916544354992 /loss/FBt4/assignment_loss‚0.801284987712279 1loss/SRqqt3/detection_loss‚1.1425767317414284 1loss/SRqqt4/detection_loss‚1.4619700616156612 %loss/total_loss‚47.87876972203272 % _timestamp‚1.7885264092288153e+09 /loss/FBt2/detection_loss‚2.4553113505244255 1loss/SRqqt2/assignment_loss‚2.286734661748332 1loss/FBt2/assignment_loss‚0.44078375247107715 0loss/FBt3/assignment_loss‚0.2771445251570199 trainer/global_step‚1499 _runtime‚1792.764094279 0loss/FRt3/assignment_loss‚1.6426180059972562 .loss/FRt2/detection_loss‚2.859665135145187 .loss/FBt1/detection_loss‚2.521925284582026 epoch‚1 _step‚122 -loss/FRt4/assignment_loss‚2.5827534755071 1loss/FBt1/assignment_loss‚0.39970160027293306 -loss/FRt1/detection_loss‚3.15922584577843 0loss/SRqqt1/detection_loss‚2.352553766507369 .loss/FBt4/detection_loss‚2.724981650710106 0loss/FRt1/assignment_loss‚3.5475177345452487 /loss/FRt2/assignment_loss‚4.752483673095703 /loss/FRt4/detection_loss‚1.8843494617690644 0loss/SRqqt2/detection_loss‚2.883831451016088 /loss/FBt3/detection_loss‚0.5646726730313286 /loss/FRt3/detection_loss‚1.5750908365375118ôË–XÓÒ -loss/FRt4/assignment_loss‚2.5827534755071 1loss/FBt1/assignment_loss‚0.39970160027293306 -loss/FRt1/detection_loss‚3.15922584577843 0loss/SRqqt1/detection_loss‚2.352553766507369 .loss/FBt4/detection_loss‚2.724981650710106 0loss/FRt1/assignment_loss‚3.5475177345452487 /loss/FRt2/assignment_loss‚4.752483673095703 /loss/FRt4/detection_loss‚1.8843494617690644 0loss/SRqqt2/detection_loss‚2.883831451016088 /loss/FBt3/detection_loss‚0.5646726730313286 /loss/FRt3/detection_loss‚1.5750908365375118 2loss/SRqqt1/assignment_loss‚2.5838875885193167 2loss/SRqqt3/assignment_loss‚1.3383138136123307 2loss/SRqqt4/assignment_loss‚1.6393916544354992 /loss/FBt4/assignment_loss‚0.801284987712279 1loss/SRqqt3/detection_loss‚1.1425767317414284 1loss/SRqqt4/detection_loss‚1.4619700616156612 %loss/total_loss‚47.87876972203272 % _timestamp‚1.7885264092288153e+09 /loss/FBt2/detection_loss‚2.4553113505244255 1loss/SRqqt2/assignment_loss‚2.286734661748332 1loss/FBt2/assignment_loss‚0.44078375247107715 0loss/FBt3/assignment_loss‚0.2771445251570199 trainer/global_step‚1499 _runtime‚1792.764094279 0loss/FRt3/assignment_loss‚1.6426180059972562 .loss/FRt2/detection_loss‚2.859665135145187 .loss/FBt1/detection_loss‚2.521925284582026 epoch‚1 _step‚122z*bJ(“Ô‚(:ˆ ÐþêÔ¢‘ôá gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÔçÕ‚(:„ ÐþêÔɵ†ã! disk.sda2.out‚171.63427734375 disk.md0.out‚ 164.15625& proc.memory.availableMB‚ 7.98046875 cpu‚25.18659795258646 proc.cpu.threads‚17 disk.rbd0.out‚ 0.62109375 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚0 disk.sda.out‚171.63427734375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.in‚0 disk.dm-0.out‚28312.34765625! disk.sdb2.out‚171.63427734375# disk.nvme0n1.out‚18950.28515625 network.sent‚ 2.657033e+06 disk.sda1.out‚0 disk.sdb.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 9362.0625 disk.md0p1.out‚ 164.15625) disk./.usagePercent‚75.91644179782215$ disk./.usageGB‚2715.139678955078 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚99.95129108428955* proc.memory.percent‚18.190503120422363 network.recv‚668861 disk.sdb1.in‚0 disk.sdb2.in‚0# proc.memory.rssMB‚ 2980.33203125 disk.sdb.out‚171.63427734375 disk.md0p1.in‚0ƒ°°/‘Ö‚(:† ßþêÔÉ’ØÞ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë@&Pׂ(:‰ ßþêÔËüÏà disk.sdb2.out‚173.0322265625 cpu‚25.185417357603157 disk.dm-0.in‚0& proc.memory.availableMB‚ 7.96484375 disk.sda.out‚173.0322265625 disk.sdb1.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb1.in‚0) disk./.usagePercent‚75.90169203281663 disk.sda2.in‚0 disk.sdb2.in‚0 disk.rbd2.out‚ 0.783203125 disk.md0.in‚0 disk.md0.out‚ 165.48046875 disk.nvme1n1.out‚ 9405.90625 disk.sda1.in‚0 disk.dm-0.out‚28513.62109375% disk./.usageGB‚2714.6121559143066 disk.rbd2.in‚0 disk.sda2.out‚173.0322265625 disk.sda.in‚0 disk.md0p1.out‚ 165.48046875 disk.rbd0.in‚0* proc.memory.percent‚18.190503120422363 network.recv‚675225# disk.nvme0n1.out‚19107.71484375 disk.sdb.in‚0 disk.nvme1n1.in‚0$ memory_percent‚99.95138645172119 network.sent‚ 2.753349e+06 disk.nvme0n1.in‚0 disk.sdb.out‚173.0322265625 disk.md0p1.in‚0 disk.rbd0.out‚ 0.62109375# proc.memory.rssMB‚ 2980.33203125_&\õ“Ø‚(:ˆ îþêÔפÃà gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚149.782' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.4455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ±0IÙ‚(:ˆ îþêÔçÏ‚â$ disk./.usageGB‚2714.567485809326 disk.sda1.out‚0 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0 disk.sdb.out‚174.31201171875 disk.sdb2.in‚0! disk.sdb2.out‚174.31201171875 disk.rbd0.in‚0) disk./.usagePercent‚75.90044303798548 disk.rbd2.in‚0! disk.nvme0n1.out‚ 19317.359375 disk.md0p1.in‚0 disk.sda.out‚174.31201171875 disk.sda2.in‚0 disk.sdb.in‚0 disk.sdb1.out‚0 network.sent‚ 2.755828e+06 disk.nvme1n1.in‚0$ memory_percent‚99.95136260986328 network.recv‚678204 disk.md0.out‚ 166.7109375 disk.dm-0.out‚28723.94921875 disk.sda.in‚0" disk.nvme1n1.out‚ 9406.58984375 disk.sdb1.in‚0* proc.memory.percent‚18.190503120422363 cpu‚25.184147234390192 disk.sda1.in‚0! disk.sda2.out‚174.31201171875 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚ 0.62109375# proc.memory.availableMB‚7.96875# proc.memory.rssMB‚ 2980.33203125 proc.cpu.threads‚17 disk.md0p1.out‚ 166.7109375Wö–ô”Ú‚(:‰ ýþêÔ³Ó ß gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åt® Û‚(:… ýþêÔ•Ýîà# proc.memory.rssMB‚ 2980.33203125 disk.sdb.in‚0) disk./.usagePercent‚75.90240420518961 disk.sdb2.in‚0# disk.nvme0n1.out‚19474.71484375 disk.sda2.out‚175.7978515625& proc.memory.availableMB‚ 7.98046875 disk.md0.out‚ 168.13671875 disk.dm-0.out‚28953.46484375 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.190503120422363 cpu‚25.182835622570728$ disk./.usageGB‚2714.637626647949 disk.dm-0.in‚0 disk.sdb2.out‚175.7978515625 disk.sda1.in‚0 network.sent‚ 2.761942e+06 network.recv‚682969 disk.sda.in‚0 disk.sda.out‚175.7978515625 disk.rbd2.in‚0$ memory_percent‚99.95129108428955 disk.sdb.out‚175.7978515625 disk.nvme1n1.in‚0 disk.nvme1n1.out‚9478.75 disk.md0p1.in‚0 disk.md0p1.out‚ 168.13671875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.in‚0 disk.rbd0.out‚ 0.62109375 V2Ü- _wandbruntime‚1847  _runtime‚1847UM¶»ÅÝ¿ & lr-AdamW/pg1‚0.0007300378912604587 & lr-AdamW/pg2‚0.0007300378912604587 trainer/global_step‚1549 $ _timestamp‚1.788526463633924e+09 _runtime‚ 1847.16946321 _step‚123تX±ÉÞà $ _timestamp‚1.788526463633924e+09 _runtime‚ 1847.16946321 _step‚123 & lr-AdamW/pg1‚0.0007300378912604587 & lr-AdamW/pg2‚0.0007300378912604587 trainer/global_step‚1549{ÔjÏH2ß- _wandbruntime‚1847  _runtime‚1847¿+ Æâ àåO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129951858968064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39165019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9411730 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788526463634938e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32233780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9412329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852304580608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145267764018688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1770130846208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852075797504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1763510165221 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32233780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131722007817216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125675950785616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39138677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28721934 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32233782 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143415712161280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127438552975841 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125675967824976JÖï'áéO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129951858968064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39165019 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9411730 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788526463634938e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32233780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9412329 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1852304580608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145267764018688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1770130846208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852075797504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1763510165221 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32233780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131722007817216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125675950785616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39138677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28721934 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32233782 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143415712161280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127438552975841 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125675967824976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0| ‘¨2â- _wandbruntime‚1848  _runtime‚1848¿âö&ãðM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220381184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19322269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39189876 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771252319744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1771252319744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 % _timestamp‚1.7885264646540892e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131803820389888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131804748376576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130033496056832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144P‚ú½äôM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131804748376576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130033496056832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220381184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19322269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39189876 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771252319744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1771252319744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 1440114#ÆG=2784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 % _timestamp‚1.7885264646540892e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131803820389888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995}ÛPx²2å- _wandbruntime‚1848  _runtime‚1848)|xzTæÎ % _timestamp‚1.7885264646560094e+09 2loss/SRqqt4/assignment_loss‚1.8086446999357297 1loss/FBt2/assignment_loss‚0.48670116026187316 0loss/FBt3/assignment_loss‚0.2616006427599931 .loss/FRt4/detection_loss‚2.012881797053577 -loss/FBt2/detection_loss‚2.26337844984872 /loss/FBt4/detection_loss‚2.4368050694465637 0loss/FRt3/assignment_loss‚1.9599990409154158 .loss/FRt2/assignment_loss‚4.47906813621521 1loss/SRqqt1/assignment_loss‚2.169636018574238 2loss/SRqqt2/assignment_loss‚1.8807021223980447 0loss/FBt4/assignment_loss‚0.4761140048503876 .loss/FRt1/detection_loss‚2.861114958354405 .loss/FRt3/detection_loss‚2.110687656328082 /loss/FBt3/detection_loss‚0.9220628987510061 0loss/FBt1/assignment_loss‚0.2339960044482666 1loss/SRqqt3/detection_loss‚1.5952072265629584 -loss/FBt1/detection_loss‚3.43490788936615 _runtime‚1848.191449065 0loss/FRt4/assignment_loss‚3.2299649952089085 2loss/SRqqt3/assignment_loss‚1.3925376708690937 1loss/SRqqt1/detection_loss‚2.6218651589006186 %loss/total_loss‚50.55027612715074 _step‚126 /loss/FRt1/assignment_loss‚4.557359201567514 .loss/FRt2/detection_loss‚2.804745035171509 1loss/SRqqt4/detection_loss‚1.4749630232078907 trainer/global_step‚1549 1loss/SRqqt2/detection_loss‚3.0753332661545794 epoch‚1˜ÊÀAXçÒ 1loss/SRqqt2/detection_loss‚3.0753332661545794 epoch‚1 % _timestamp‚1.7885264646560094e+09 2loss/SRqqt4/assignment_loss‚1.8086446999357297 1loss/FBt2/assignment_loss‚0.48670116026187316 0loss/FBt3/assignment_loss‚0.2616006427599931 .loss/FRt4/detection_loss‚2.012881797053577 -loss/FBt2/detection_loss‚2.26337844984872 /loss/FBt4/detection_loss‚2.4368050694465637 0loss/FRt3/assignment_loss‚1.9599990409154158 .loss/FRt2/assignment_loss‚4.47906813621521 1loss/SRqqt1/assignment_loss‚2.169636018574238 2loss/SRqqt2/assignment_loss‚1.8807021223980447 0loss/FBt4/assignment_loss‚0.4761140048503876 .loss/FRt1/detection_loss‚2.861114958354405 .loss/FRt3/detection_loss‚2.110687656328082 /loss/FBt3/detection_loss‚0.9220628987510061 0loss/FBt1/assignment_loss‚0.2339960044482666 1loss/SRqqt3/detection_loss‚1.5952072265629584 -loss/FBt1/detection_loss‚3.43490788936615 _runtime‚1848.191449065 0loss/FRt4/assignment_loss‚3.2299649952089085 2loss/SRqqt3/assignment_loss‚1.3925376708690937 1loss/SRqqt1/detection_loss‚2.6218651589006186 %loss/total_loss‚50.55027612715074 _step‚126 /loss/FRt1/assignment_loss‚4.557359201567514 .loss/FRt2/detection_loss‚2.804745035171509 1loss/SRqqt4/detection_loss‚1.4749630232078907 trainer/global_step‚1549~Y ”è‚(:‰ ŒÿêÔÝ©Êá gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬“ê­ é‚(:‚ ŒÿêÔÎâã disk.sdb2.in‚0! disk.sdb2.out‚177.17724609375 disk.nvme1n1.out‚ 9506.984375 disk.md0p1.out‚ 169.4609375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 2980.33203125 cpu‚25.181585159136386$ disk./.usageGB‚2714.670928955078 disk.sda.in‚0 network.sent‚ 2.78115e+06 disk.md0p1.in‚0 disk.rbd0.out‚ 0.62109375* proc.memory.percent‚18.190503120422363 proc.cpu.threads‚17 network.recv‚687507! disk.nvme0n1.out‚ 19631.953125# memory_percent‚99.9514102935791 disk.rbd0.in‚0 disk.rbd2.out‚ 0.783203125! disk.sda2.out‚177.17724609375 disk.sda.out‚177.17724609375 disk.sdb1.out‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 7.9609375) disk./.usagePercent‚75.90333535163502 disk.sda2.in‚0 disk.md0.in‚0 disk.md0.out‚ 169.4609375 disk.dm-0.out‚ 29138.9375 disk.nvme1n1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0 disk.sdb.out‚177.17724609375[›Ôœ’ê‚(:‡ ›ÿêÔú›«à gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m»üÁë‚(:‰ ›ÿêÔÚ ™á disk.nvme1n1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚19841.69921875) disk./.usagePercent‚75.90946470577393 disk.rbd0.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb1.in‚0 disk.sda2.out‚178.7333984375 network.recv‚693534 disk.dm-0.out‚ 29400.7109375 disk.sdb2.out‚178.7333984375 disk.md0p1.out‚ 170.94921875 disk.sdb1.out‚0 disk.rbd2.out‚ 0.783203125 proc.cpu.threads‚17 network.sent‚ 2.875079e+06 disk.sdb.in‚0" disk.nvme1n1.out‚ 9559.01171875 disk.md0p1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2980.33203125* proc.memory.percent‚18.190503120422363 disk.sda.out‚178.7333984375 disk.sda2.in‚0& proc.memory.availableMB‚ 8.11328125% disk./.usageGB‚2714.8901443481445 disk.dm-0.in‚0 disk.md0.out‚ 170.94921875 cpu‚25.18046452827043 disk.sdb.out‚178.7333984375 disk.sda.in‚0 disk.md0.in‚0# memory_percent‚99.9504804611206 disk.sdb2.in‚0 disk.nvme0n1.in‚0¥¢÷³”ì‚(:‰ ªÿêÔ‚±Áà gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l#ÍMí‚(:ƒ ªÿêÔ•À€â# memory_percent‚99.9504804611206* proc.memory.percent‚18.190503120422363 disk.sdb1.out‚0 disk.sda2.out‚180.7177734375 disk.sdb2.out‚180.7177734375 disk.sda.in‚0 disk.sdb2.in‚0# proc.memory.rssMB‚ 2980.33203125 cpu‚25.17928944462912 disk.nvme1n1.in‚0) disk./.usagePercent‚75.90945169318901 disk.rbd0.out‚ 0.62109375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.out‚ 19998.9375 proc.cpu.threads‚17 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚0 disk.md0.in‚0 disk.sdb.out‚180.7177734375 disk.md0.out‚ 172.86328125 disk.dm-0.in‚0 disk.dm-0.out‚29635.62890625& proc.memory.availableMB‚ 8.11328125 network.sent‚ 2.8828e+06 network.recv‚697632 disk.sda1.out‚0" disk.nvme1n1.out‚ 9636.69140625 disk.md0p1.in‚0$ disk./.usageGB‚2714.889678955078 disk.sdb1.in‚0 disk.sda.out‚180.7177734375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚0 disk.md0p1.out‚ 172.86328125Ù̹Ž2î- _wandbruntime‚1902  _runtime‚1902b=S¾ÇïÁ % _timestamp‚1.7885265188456774e+09 _runtime‚1902.381138793 _step‚127 & lr-AdamW/pg1‚0.0007294822696082292 & lr-AdamW/pg2‚0.0007294822696082292 trainer/global_step‚1599Umå¬ËðÅ _step‚127 & lr-AdamW/pg1‚0.0007294822696082292 & lr-AdamW/pg2‚0.0007294822696082292 trainer/global_step‚1599 % _timestamp‚1.7885265188456774e+09 _runtime‚1902.381138793A¼‹­2ñ- _wandbruntime‚1902  _runtime‚1902îJÃPì'òæO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134032831262208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚1599 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149837740649472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135859049533440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40383579 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134032813259264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826218271232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134032813259264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131445309958149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40409921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131446234972505 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147926973443584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129626847552176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147926997383680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826218271232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134032813259264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131445309958149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40409921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131446234972505 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147926973443584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129626847552176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147926997383680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826218271232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19925519 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9707272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1911925198336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134114450348032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40408410 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1819596718611 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134114432345088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29632793 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135941790092800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149928899770880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131526369840535 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9706652 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827339744768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73670350 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33261942 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131525444825971 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚1903.501726439 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149929148286464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1911700622848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73670350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40434778 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19925519 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9707272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1911925198336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134114450348032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40408410 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1819596718611 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134114432345088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29632793 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135941790092800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149928899770880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131526369840535 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9706652 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827339744768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73670350 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33261942 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131525444825971 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚1903.501726439 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149929148286464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1911700622848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73670350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247yYÙ*2÷- _wandbruntime‚1903  _runtime‚1903‘”ÑrWøÑ trainer/global_step‚1599 2loss/FBt4/assignment_loss‚4.51792984677013e-05 0loss/FRt2/assignment_loss‚4.8928676545619965 /loss/FRt3/assignment_loss‚1.819464038400089 2loss/SRqqt1/assignment_loss‚1.8747597302381809 .loss/FRt4/detection_loss‚1.907815516394164 0loss/SRqqt2/detection_loss‚2.734145601987839 1loss/SRqqt3/detection_loss‚1.4129344990192834 0loss/FBt2/assignment_loss‚0.6511351430747092 .loss/FRt2/detection_loss‚2.606263056397438 .loss/FRt3/detection_loss‚1.852879787338715 .loss/FBt1/detection_loss‚2.690252657979727 /loss/FBt2/detection_loss‚2.7383946435792104 0loss/FRt4/assignment_loss‚2.5925883459193364 /loss/FBt3/detection_loss‚0.9075726645144945 -loss/FBt4/detection_loss‚2.70384019613266 $loss/total_loss‚47.7535723100264 1loss/SRqqt2/assignment_loss‚2.056293935775757 2loss/SRqqt3/assignment_loss‚1.0483960021066212 1loss/FBt1/assignment_loss‚0.16025185640432937 0loss/FBt3/assignment_loss‚0.2375613066382152 _runtime‚1903.502766464 _step‚130 0loss/FRt1/assignment_loss‚4.7312435167176385 /loss/FRt1/detection_loss‚2.9963296119655882 epoch‚1 0loss/SRqqt1/detection_loss‚2.599973121514687 % _timestamp‚1.7885265199675024e+09 2loss/SRqqt4/assignment_loss‚1.3423270154149924 1loss/SRqqt4/detection_loss‚1.1962372286522642¦7|\ùÖ 0loss/FRt2/assignment_loss‚4.8928676545619965 /loss/FRt3/assignment_loss‚1.819464038400089 2loss/SRqqt1/assignment_loss‚1.8747597302381809 .loss/FRt4/detection_loss‚1.907815516394164 0loss/SRqqt2/detection_loss‚2.734145601987839 1loss/SRqqt3/detection_loss‚1.4129344990192834 0loss/FBt2/assignment_loss‚0.6511351430747092 .loss/FRt2/detection_loss‚2.606263056397438 .loss/FRt3/detection_loss‚1.852879787338715 .loss/FBt1/detection_loss‚2.690252657979727 /loss/FBt2/detection_loss‚2.7383946435792104 0loss/FRt4/assignment_loss‚2.5925883459193364 /loss/FBt3/detection_loss‚0.9075726645144945 -loss/FBt4/detection_loss‚2.70384019613266 $loss/total_loss‚47.7535723100264 1loss/SRqqt2/assignment_loss‚2.056293935775757 2loss/SRqqt3/assignment_loss‚1.0483960021066212 1loss/FBt1/assignment_loss‚0.16025185640432937 0loss/FBt3/assignment_loss‚0.2375613066382152 _runtime‚1903.502766464 _step‚130 0loss/FRt1/assignment_loss‚4.7312435167176385 /loss/FRt1/detection_loss‚2.9963296119655882 epoch‚1 0loss/SRqqt1/detection_loss‚2.599973121514687 % _timestamp‚1.7885265199675024e+09 2loss/SRqqt4/assignment_loss‚1.3423270154149924 1loss/SRqqt4/detection_loss‚1.1962372286522642 trainer/global_step‚1599 2loss/FBt4/assignment_loss‚4.51792984677013e-05‚ço妓ú‚(:ˆ ¹ÿêÔÔ¡¬á gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚138.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.6255 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xý+û‚(:ú ¹ÿêÔöŽÅá" disk.nvme1n1.out‚ 9714.46484375 disk.rbd0.in‚0) disk./.usagePercent‚75.91647027618423 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2980.33203125 cpu‚25.17803719466451 disk.md0.out‚ 174.15234375 disk.sdb2.in‚0 disk.sda.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚0$ proc.memory.availableMB‚7.859375$ disk./.usageGB‚2715.140697479248 disk.md0.in‚0 disk.sdb.out‚ 182.0703125 disk.md0p1.in‚0 disk.md0p1.out‚ 174.15234375 disk.sda1.out‚0 network.sent‚ 2.886738e+06 network.recv‚702149 disk.sda.out‚ 182.0703125 disk.rbd0.out‚ 0.62109375# disk.nvme0n1.out‚19999.01953125 disk.sda2.in‚0* proc.memory.percent‚18.190503120422363 disk.dm-0.out‚ 29713.484375 disk.rbd2.out‚ 0.783203125 disk.sda2.out‚ 182.0703125$ memory_percent‚99.95203018188477 disk.nvme0n1.in‚0 disk.dm-0.in‚0 proc.cpu.threads‚17 disk.sdb2.out‚ 182.0703125\>阓ü‚(:ˆ ÈÿêÔ¿ä¯à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê{üý‚(:ñ ÈÿêÔ®…ãá disk.sda1.out‚0 disk.sda.in‚0 disk.md0p1.out‚ 175.41015625" proc.memory.rssMB‚ 2980.3359375$ disk./.usageGB‚2715.139678955078 disk.sdb1.out‚0 disk.md0.in‚0 proc.cpu.threads‚17 network.recv‚709146 disk.dm-0.in‚0 disk.sdb.in‚0 disk.md0p1.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚ 183.40625 disk.sdb.out‚ 183.40625 disk.sdb2.out‚ 183.40625 disk.rbd0.out‚ 0.62109375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda.out‚ 183.40625$ memory_percent‚99.95079040527344 cpu‚25.176914312967025 network.sent‚ 2.996445e+06) disk./.usagePercent‚75.91644179782215" disk.nvme0n1.out‚ 20156.2578125 disk.md0.out‚ 175.41015625 disk.dm-0.out‚29948.39453125 disk.sdb2.in‚0 disk.rbd0.in‚0" proc.memory.availableMB‚8.0625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚18.190526962280273" disk.nvme1n1.out‚ 9792.13671875b9Z“þ‚(:ˆ ×ÿêÔ£ñþß gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sz1iÿ‚(:† ×ÿêÔ½–Õà disk.rbd0.out‚ 0.62109375 disk.sdb.in‚0 disk.sda.out‚184.84521484375 proc.cpu.threads‚17 disk.md0.out‚ 176.7734375$ disk./.usageGB‚2715.139678955078 disk.sdb.out‚184.84521484375# memory_percent‚99.9500036239624 network.sent‚ 2.999673e+06 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 20418.3984375! disk.sda2.out‚184.84521484375 disk.md0.in‚0) disk./.usagePercent‚75.91644179782215 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 30314.0390625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 176.7734375 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.190526962280273 network.recv‚713032! disk.sdb2.out‚184.84521484375 disk.sda.in‚0 disk.nvme1n1.out‚ 9895.640625 disk.md0p1.in‚0& proc.memory.availableMB‚ 8.19140625 disk.sda1.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2980.3359375 cpu‚25.17571693992361 disk.sda2.in‚0 disk.dm-0.in‚0§À÷’€‚(:‡ æÿêÔ͉ à gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†·,Æ‚(:û æÿêԪزá disk.rbd0.in‚0 disk.rbd0.out‚ 0.62109375 network.recv‚716783 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda.out‚ 186.576171875 disk.nvme1n1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.in‚0 disk.sda.in‚0$ memory_percent‚99.95384216308594 disk.md0p1.out‚ 178.4296875 disk.md0.out‚ 178.4296875" disk.nvme0n1.out‚ 20479.2890625 disk.sda2.in‚0 disk.md0p1.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 186.576171875 disk.dm-0.in‚0" disk.nvme1n1.out‚ 10016.7734375 disk.sdb.out‚ 186.576171875 disk.sdb2.in‚0 disk.sdb2.out‚ 186.576171875" proc.memory.availableMB‚7.5625" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 3.005156e+06) disk./.usagePercent‚75.90169107287184% disk./.usageGB‚2714.6121215820312 disk.nvme0n1.in‚0 disk.dm-0.out‚ 30482.0625 disk.sdb1.in‚0* proc.memory.percent‚18.190526962280273 cpu‚25.17474216067147 disk.md0.in‚0èiÊ 2‚- _wandbruntime‚1957  _runtime‚1957ÝqÔ9Ń¿ # _timestamp‚1.78852657417047e+09 _runtime‚1957.706048806 _step‚131 & lr-AdamW/pg1‚0.0007288508779012518 & lr-AdamW/pg2‚0.0007288508779012518 trainer/global_step‚1649ücDýÊ„Ä & lr-AdamW/pg1‚0.0007288508779012518 & lr-AdamW/pg2‚0.0007288508779012518 trainer/global_step‚1649 # _timestamp‚1.78852657417047e+09 _runtime‚1957.706048806 _step‚131ƒ¨ 2…- _wandbruntime‚1957  _runtime‚1957OÅÆòÏ †æO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1875264675437 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1882305696256 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139995163276288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1874356700441 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_tÙ õrain_batch_start/allocation.all.allocated‚75904605 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34249780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75878261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚1957.707276801 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139996091249664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152438258665984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1882305696256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41628481 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1968950503424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152438282606080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20516704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41654823 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135452066940457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133577710240016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚132 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30509447 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139996091249664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138113785553408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ç½øñ'‡ëO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34249780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75878261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚1957.707276801 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139996091249664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152438258665984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1882305696256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41628481 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1968950503424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152438282606080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20516704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41654823 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135452066940457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133577710240016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚132 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30509447 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139996091249664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138113785553408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1875264675437 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1882305696256 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139995163276288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1874356700441 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75904605 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140077903822336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75923252 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34269942 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1970351963136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133656744873920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154498603369472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚133 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1883427169792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41679680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138195404639232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41653312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1882517186048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138195386636288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 % _timestamp‚1.7885265753157067e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30527293 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1876381948923 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75923252 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1882517186048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34269940 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75949622 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34269940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247|ÅÄ\§ŠõM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75949622 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34269940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41679680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232964096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140077903822336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75923252 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34269942 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1970351963136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133656744873920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154498603369472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚133 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1883427169792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41679680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138195404639232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41653312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1882517186048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138195386636288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 % _timestamp‚1.7885265753157067e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30527293 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1876381948923 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75923252 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1882517186048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34269940 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0…UFUÎ2‹- _wandbruntime‚1958  _runtime‚1958xAE…VŒÐ 1loss/FBt4/assignment_loss‚0.29048728151246905 /loss/FRt2/detection_loss‚2.6064377633424907 -loss/FRt4/detection_loss‚2.17617632049535 _step‚134 /loss/FRt3/assignment_loss‚1.618580574989319 0loss/FRt4/assignment_loss‚3.0483429389340535 0loss/SRqqt2/assignment_loss‚1.68554022292728 2loss/SRqqt3/assignment_loss‚1.2067997237543264 0loss/FBt1/assignment_loss‚0.4104252280482494 0loss/FBt2/assignment_loss‚0.5085286604310438 .loss/FRt1/detection_loss‚2.831076034477779 epoch‚1 2loss/SRqqt4/assignment_loss‚1.4328029510432057 1loss/FBt3/assignment_loss‚0.20018972269041455 /loss/FBt1/detection_loss‚2.8132912069559097 /loss/FRt2/assignment_loss‚4.998968958854675 1loss/SRqqt1/detection_loss‚2.6935461497306825 0loss/SRqqt4/detection_loss‚1.434361292315381 2loss/SRqqt1/assignment_loss‚2.1337556746602058 /loss/SRqqt2/detection_loss‚2.61164207432581 .loss/FBt2/detection_loss‚2.508957034775189 /loss/FBt3/detection_loss‚0.8080162924906129 trainer/global_step‚1649 % _timestamp‚1.7885265753190405e+09 /loss/FRt3/detection_loss‚1.8860695651173591 1loss/SRqqt3/detection_loss‚1.1901302026212215 /loss/FBt4/detection_loss‚3.9248622059822083 %loss/total_loss‚49.53219773678191 _runtime‚1958.854375016 0loss/FRt1/assignment_loss‚4.5132096563066755…¢| [Õ /loss/SRqqt2/detection_loss‚2.61164207432581 .loss/FBt2/detection_loss‚2.508957034775189 /loss/FBt3/detection_loss‚0.8080162924906129 trainer/global_step‚1649 % _timestamp‚1.7885265753190405e+09 /loss/FRt3/detection_loss‚1.8860695651173591 1loss/SRqqt3/detection_loss‚1.1901302026212215 /loss/FBt4/detection_loss‚3.9248622059822083 %loss/total_loss‚49.53219773678191 _runtime‚1958.854375016 0loss/FRt1/assignment_loss‚4.5132096563066755 1loss/FBt4/assignment_loss‚0.29048728151246905 /loss/FRt2/detection_loss‚2.6064377633424907 -loss/FRt4/detection_loss‚2.17617632049535 _step‚134 /loss/FRt3/assignment_loss‚1.618580574989319 0loss/FRt4/assignment_loss‚3.0483429389340535 0loss/SRqqt2/assignment_loss‚1.68554022292728 2loss/SRqqt3/assignment_loss‚1.2067997237543264 0loss/FBt1/assignment_loss‚0.4104252280482494 0loss/FBt2/assignment_loss‚0.5085286604310438 .loss/FRt1/detection_loss‚2.831076034477779 epoch‚1 2loss/SRqqt4/assignment_loss‚1.4328029510432057 1loss/FBt3/assignment_loss‚0.20018972269041455 /loss/FBt1/detection_loss‚2.8132912069559097 /loss/FRt2/assignment_loss‚4.998968958854675 1loss/SRqqt1/detection_loss‚2.6935461497306825 0loss/SRqqt4/detection_loss‚1.434361292315381 2loss/SRqqt1/assignment_loss‚2.1337556746602058†AêÂz”Ž‚(:‰ õÿêÔò²ðá gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€BY­N‚(:à õÿêÔ¦§ýâ disk.sdb.out‚333.60888671875 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚99.95119571685791 disk.sda.out‚333.60888671875# disk.nvme0n1.out‚20744.45703125 disk.md0.out‚ 325.3671875 disk.dm-0.out‚30762.05078125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 325.3671875 disk.sda1.out‚0! disk.sda2.out‚333.60888671875* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 network.sent‚ 3.011371e+06) disk./.usagePercent‚75.90045551726774 disk.rbd0.out‚ 0.62109375 disk.rbd2.in‚0& proc.memory.availableMB‚ 7.99609375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5703125 cpu‚25.173508214490294% disk./.usageGB‚2714.5679321289062 disk.nvme1n1.out‚ 10017.59375 disk.md0p1.in‚ 199.5703125 disk.sda2.in‚ 128.609375" proc.memory.rssMB‚ 2980.3359375 network.recv‚721674 disk.dm-0.in‚0! disk.sdb2.out‚333.60888671875 disk.sda.in‚ 128.609375 disk.rbd2.out‚ 0.783203125 F®Ì“‚(:ˆ „€ëÔÖŽß gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.718' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.6795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿu½üI‘‚(:¾ „€ëÔ‘áßà network.recv‚727984 disk.dm-0.out‚ 31032.5703125 disk.sdb.out‚ 336.060546875 disk.sda.out‚ 336.060546875 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚ 128.609375 disk.md0.out‚ 327.74609375 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5703125 disk.rbd2.out‚ 0.783203125* proc.memory.percent‚18.190526962280273 cpu‚25.172393180983292 proc.cpu.threads‚17% disk./.usageGB‚2714.6375770568848 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 336.060546875 disk.sda.in‚ 128.609375 disk.sda1.out‚0$ memory_percent‚99.95274543762207) disk./.usagePercent‚75.90240281860268 disk.dm-0.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.sda2.out‚ 336.060546875# disk.nvme1n1.out‚10089.64453125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 327.74609375 disk.rbd2.in‚0 network.sent‚ 3.119005e+06 disk.md0.in‚ 199.5703125% proc.memory.availableMB‚ 7.7421875 disk.sdb2.in‚ 70.9609375# disk.nvme0n1.out‚20942.92578125" proc.memory.rssMB‚ 2980.3359375Æ– ”’‚(:‰ “€ëÔç’ß gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.50975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëX ^L“‚(:Á “€ëÔƒÙãà disk.md0p1.out‚ 588.703125 disk.md0.in‚ 199.57421875 disk.md0.out‚ 588.703125! disk.sdb2.out‚597.12451171875 proc.memory.availableMB‚7.75 proc.cpu.threads‚17 network.recv‚731922$ disk./.usageGB‚2714.670883178711 disk.rbd0.in‚0! disk.sda2.out‚597.12451171875 cpu‚25.17148293191164 disk.sda1.in‚0 disk.md0p1.in‚ 199.57421875 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 network.sent‚ 3.124367e+06 disk.rbd2.out‚ 0.783203125 disk.sda1.out‚0# disk.nvme1n1.out‚10117.88671875 disk.dm-0.in‚0 disk.sda.in‚ 128.61328125 disk.sda.out‚597.12451171875 disk.sdb1.in‚0# disk.nvme0n1.out‚20943.02734375 disk.sda2.in‚ 128.61328125 disk.dm-0.out‚ 31060.9140625 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.62109375 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sdb.out‚597.12451171875$ memory_percent‚99.95269775390625" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273) disk./.usagePercent‚75.903334071708640òq””‚(:‰ ¢€ëÔÁÜÇà gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë: ßC•‚(:¸ ¢€ëÔÔ³…â disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚10169.70703125" proc.memory.availableMB‚7.8125 proc.cpu.threads‚17 disk.sda.out‚717.59814453125 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.sdb.out‚717.59814453125 disk.dm-0.out‚31269.97265625 disk.sda1.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2980.3359375 disk.rbd0.in‚0 disk.nvme0n1.in‚0! disk.sda2.out‚717.59814453125 disk.md0.out‚ 709.1015625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 cpu‚25.170308558134714% disk./.usageGB‚2714.8884315490723 disk.md0p1.out‚ 709.1015625! disk.nvme0n1.out‚ 21100.265625! disk.sdb2.out‚717.59814453125 disk.sda.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 0.783203125 network.sent‚ 3.132207e+06 network.recv‚735759' disk./.usagePercent‚75.909416815195 disk.md0p1.in‚ 199.5859375 disk.md0.in‚ 199.5859375$ memory_percent‚99.95231628417969 disk.rbd0.out‚ 0.62109375* proc.memory.percent‚18.190526962280273yy>2–- _wandbruntime‚2013  _runtime‚2013¦øÇ—Á & lr-AdamW/pg1‚0.0007281438477206116 & lr-AdamW/pg2‚0.0007281438477206116 trainer/global_step‚1699 % _timestamp‚1.7885266298771172e+09 _runtime‚2013.412728751 _step‚135,z7VÌ˜Æ & lr-AdamW/pg1‚0.0007281438477206116 & lr-AdamW/pg2‚0.0007281438477206116 trainer/global_step‚1699 % _timestamp‚1.7885266298771172e+09 _runtime‚2013.412728751 _step‚135‡œ®ô…2™- _wandbruntime‚2013  _runtime‚2013} ùì'šæO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1938393121280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 _runtime‚ 2013.4147923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35257782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158977015410688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1931141930545 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158977272328192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156949543888384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2027471522304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31404482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21119954 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137528607006576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885266298788645e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930233955549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35257780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42873383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21119956 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139458823922765 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156949567828480 trainer/global_step‚1699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142194739844608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144132204992512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42899725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513<Á/ó!›ëO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930233955549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35257780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42873383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21119956 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139458823922765 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156949567828480 trainer/global_step‚1699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142194739844608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144132204992512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42899725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1938393121280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 _runtime‚ 2013.4147923 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35257782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158977015410688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1931141930545 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158977272328192 aMDeviceStatsMonitor.on_train_batch_¤b"Wþstart/inactive_split_bytes.large_pool.freed‚156949543888384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2027471522304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31404482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21119954 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137528607006576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885266298788645e+09ˆ0%2œ- _wandbruntime‚2014  _runtime‚2014Y ªœö&ðM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35277942 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35277940 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157039769592832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157039793532928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144215873525248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 # _timestamp‚1.78852663099818e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1939514594816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78202524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10289557 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137607624601120 _runtime‚2014.535494534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1938604611072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42924582 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159068658009600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚137 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚1699 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139539883805151 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10290175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2028635706880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144214945538560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35277940 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78176154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42898214 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368"Í.1û&žõM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35277942 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35277940 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157039769592832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157039793532928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144215873525248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 # _timestamp‚1.78852663099818e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1939514594816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78202524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10289557 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137607624601120 _runtime‚2014.535494534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1938604611072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42924582 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159068658009600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚137 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚1699 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139539883805151 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10290175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2028635706880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144214945538560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35277940 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78176154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42898214 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564‰Dsæo2Ÿ- _wandbruntime‚2014  _runtime‚2014 €&„Z Ô /loss/FRt1/assignment_loss‚3.817664121326647 1loss/FBt2/assignment_loss‚0.18571421752357903 /loss/FBt2/detection_loss‚2.0434472229745655 /loss/FRt2/assignment_loss‚4.278916563306536 /loss/SRqqt1/detection_loss‚2.68500123100896 0loss/SRqqt2/detection_loss‚2.395435890247082 0loss/FBt4/assignment_loss‚0.1043051965534687 /loss/FRt3/detection_loss‚1.7105967754291163 0loss/FRt3/assignment_loss‚1.7444367298373469 /loss/FRt4/assignment_loss‚2.390245111621156 .loss/FBt1/detection_loss‚4.366920018196106 _step‚138 2loss/SRqqt1/assignment_loss‚1.6986310357047665 2loss/SRqqt3/assignment_loss‚1.2866977966870323 2loss/SRqqt4/assignment_loss‚1.3534806193897455 1loss/FBt1/assignment_loss‚0.29000015768942833 1loss/SRqqt4/detection_loss‚1.3789342741777257 1loss/FBt3/assignment_loss‚0.15150806297737537 1loss/SRqqt3/detection_loss‚1.3129220457013808 /loss/FBt3/detection_loss‚0.7657090696494704 /loss/FRt2/detection_loss‚2.9217910511153087 2loss/SRqqt2/assignment_loss‚1.7276277598635903 %loss/total_loss‚47.09800319802059 trainer/global_step‚1699 $ _timestamp‚1.788526631002128e+09 /loss/FRt1/detection_loss‚3.4512194269581844 /loss/FRt4/detection_loss‚2.0675253959334627 /loss/FBt4/detection_loss‚2.9692734241485597 epoch‚1 _runtime‚2014.537709424­P×_¡Ù /loss/FRt1/assignment_loss‚3.817664121326647 1loss/FBt2/assignment_loss‚0.18571421752357903 /loss/FBt2/detection_loss‚2.0434472229745655 /loss/FRt2/assignment_loss‚4.278916563306536 /loss/SRqqt1/detection_loss‚2.68500123100896 0loss/SRqqt2/detection_loss‚2.395435890247082 0loss/FBt4/assignment_loss‚0.1043051965534687 /loss/FRt3/detection_loss‚1.7105967754291163 0loss/FRt3/assignment_loss‚1.7444367298373469 /loss/FRt4/assignment_loss‚2.390245111621156 .loss/FBt1/detection_loss‚4.366920018196106 _step‚138 2loss/SRqqt1/assignment_loss‚1.6986310357047665 2loss/SRqqt3/assignment_loss‚1.2866977966870323 2loss/SRqqt4/assignment_loss‚1.3534806193897455 1loss/FBt1/assignment_loss‚0.29000015768942833 1loss/SRqqt4/detection_loss‚1.3789342741777257 1loss/FBt3/assignment_loss‚0.15150806297737537 1loss/SRqqt3/detection_loss‚1.3129220457013808 /loss/FBt3/detection_loss‚0.7657090696494704 /loss/FRt2/detection_loss‚2.9217910511153087 2loss/SRqqt2/assignment_loss‚1.7276277598635903 %loss/total_loss‚47.09800319802059 trainer/global_step‚1699 $ _timestamp‚1.788526631002128e+09 /loss/FRt1/detection_loss‚3.4512194269581844 /loss/FRt4/detection_loss‚2.0675253959334627 /loss/FBt4/detection_loss‚2.9692734241485597 epoch‚1 _runtime‚2014.537709424Šr×3~‘¢‚(:† ±€ëԧОá gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô–«ÊH£‚(:½ ±€ëÔàÄâ disk.sdb.out‚718.64892578125 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚718.64892578125 disk.sdb1.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 network.recv‚740514$ disk./.usageGB‚2714.889820098877 disk.sda2.in‚128.625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 21310.1015625 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.md0p1.out‚ 710.1015625$ memory_percent‚99.95248317718506 disk.dm-0.in‚0 disk.dm-0.out‚31635.28515625# disk.nvme1n1.out‚10325.18359375" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273( disk./.usagePercent‚75.9094556396287 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0! disk.sda2.out‚718.64892578125 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.sda.out‚718.64892578125 disk.sda1.in‚0 disk.md0.out‚ 710.1015625 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0& proc.memory.availableMB‚ 7.78515625 cpu‚25.169306022179775 network.sent‚ 3.149097e+06á-W“¤‚(:ˆ À€ëÔöã¥à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿŠó­>¥‚(:³ À€ëÔ¢ëùà network.recv‚747511 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 0.62109375" disk.nvme0n1.out‚ 21467.3359375$ memory_percent‚99.95386600494385& proc.memory.availableMB‚ 7.55859375 cpu‚25.16834490595791 disk.dm-0.out‚31840.68359375 disk.md0p1.out‚711.375 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0! disk.sdb2.out‚719.97509765625 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625* proc.memory.percent‚18.190526962280273( disk./.usagePercent‚75.9164742226239 disk.sdb.out‚719.97509765625 disk.sda.out‚719.97509765625 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚10373.34765625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 0.783203125 disk.md0.out‚711.375! disk.sda2.out‚719.97509765625$ disk./.usageGB‚2715.140838623047 disk.sdb1.in‚0 network.sent‚ 3.24622e+06 disk.sda.in‚128.625 disk.sdb1.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2980.3359375`2’¦‚(:‡ Ï€ëÔ™ˆŠß gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¯'äH§‚(:½ Ï€ëÔïšà disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.190526962280273$ disk./.usageGB‚2715.139820098877 disk.sda1.out‚0" disk.nvme1n1.out‚ 10454.6953125 disk.md0p1.out‚ 713.01171875& proc.memory.availableMB‚ 7.41015625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375" proc.memory.rssMB‚ 2980.3359375 cpu‚25.167389555734477 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚721.67919921875 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚99.95477199554443 proc.cpu.threads‚17 network.sent‚ 3.249382e+06 disk.rbd0.in‚0 disk.dm-0.out‚ 32131.734375 disk.nvme1n1.in‚0" disk.nvme0n1.out‚ 21677.0390625 disk.sdb.out‚721.67919921875 disk.sda.out‚721.67919921875 network.recv‚751260 disk.sda.in‚128.625) disk./.usagePercent‚75.91644574426184 disk.sdb1.in‚0! disk.sda2.out‚721.67919921875 disk.md0.out‚ 713.01171875›qy=”¨‚(:‰ Þ€ëÔü¹Àß gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éå3õG©‚(:¼ Þ€ëÔžÄËà disk.sdb2.out‚723.2001953125 disk.sda2.out‚723.2001953125 disk.sdb.in‚ 70.9609375 disk.md0.out‚ 714.46484375& proc.memory.availableMB‚ 7.41796875 cpu‚25.166539292278596 disk.sda.out‚723.2001953125 disk.md0p1.in‚ 199.5859375# disk.nvme0n1.out‚21834.27734375" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 disk.dm-0.out‚32418.26953125 disk.sdb1.out‚0 disk.sdb.out‚723.2001953125 disk.md0p1.out‚ 714.46484375 disk.sdb1.in‚0 disk.nvme0n1.in‚0$ memory_percent‚99.95472431182861 proc.cpu.threads‚17 network.recv‚754909 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.62109375 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 network.sent‚ 3.254998e+06) disk./.usagePercent‚75.91644574426184 disk.dm-0.in‚0" disk.nvme1n1.out‚ 10583.9921875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚128.625$ disk./.usageGB‚2715.139820098877+2ª- _wandbruntime‚2070  _runtime‚2070L¶ÓÇ«Á % _timestamp‚1.7885266867654183e+09 _runtime‚2070.300969359 _step‚139 & lr-AdamW/pg1‚0.0007273613264103389 & lr-AdamW/pg2‚0.0007273613264103389 trainer/global_step‚1749i^ÄÌ¬Æ & lr-AdamW/pg1‚0.0007273613264103389 & lr-AdamW/pg2‚0.0007273613264103389 trainer/global_step‚1749 % _timestamp‚1.7885266867654183e+09 _runtime‚2070.300969359 _step‚139‹åÄq12­- _wandbruntime‚2070  _runtime‚2070ÚsC®èO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146275694135808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148270174682112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36265782 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10574723 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163546979465728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146275676132864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143465580905073 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_p6£˜ÉÜ ool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80384065 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚140 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21723206 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163547217508864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2086150354944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1994480546304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1993570575872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885266867661538e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36265782 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44118285 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146275676132864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚1749 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80410409 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1987019185653 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10574117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143465580905073 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80384065 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚140 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21723206 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163547217508864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2086150354944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1994480546304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1993570575872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885266867661538e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36265782 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44118285 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146275676132864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚1749 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80410409 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1987019185653 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10574117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36285942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44143116 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44143116 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80455426 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44169484 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2087539679232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148352915241472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148351987254784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10580495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚141 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44169484 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640çÑä²÷M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10580495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚141 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44169484 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceSt ¾SatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163638359433216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146357313221632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1987228483935 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36285942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44143116 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44143116 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80455426 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44169484 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2087539679232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148352915241472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148351987254784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560¾{2³- _wandbruntime‚2071  _runtime‚2071µÓê5\´Ö 1loss/FBt1/assignment_loss‚0.32845896511560585 /loss/FBt3/detection_loss‚0.9695379601255285 trainer/global_step‚1749 _runtime‚2071.319556407 _step‚142 /loss/FRt2/assignment_loss‚5.138735751310985 2loss/SRqqt1/assignment_loss‚2.4582864741484323 /loss/FBt4/detection_loss‚3.0600304901599884 epoch‚1 .loss/FBt2/detection_loss‚2.543050597111384 1loss/SRqqt2/assignment_loss‚1.515948401298374 /loss/FRt3/detection_loss‚1.6645975448191166 /loss/FRt4/detection_loss‚1.7459372490262375 1loss/SRqqt1/detection_loss‚2.2301406876908407 % _timestamp‚1.7885266877842543e+09 /loss/FRt1/assignment_loss‚4.746986071268718 0loss/FBt2/assignment_loss‚0.3793385189481569 0loss/FBt4/assignment_loss‚0.2922340748482384 0loss/FRt3/assignment_loss‚1.5473963675590663 0loss/FRt4/assignment_loss‚2.6584985508368564 2loss/SRqqt3/assignment_loss‚1.1491134388923225 /loss/FRt2/detection_loss‚2.7888467450936636 /loss/FBt1/detection_loss‚2.4971117774645486 %loss/total_loss‚47.40253436271518 /loss/FRt1/detection_loss‚2.8553469445970325 2loss/SRqqt4/assignment_loss‚1.3874578049809143 1loss/FBt3/assignment_loss‚0.29939049651875693 0loss/SRqqt2/detection_loss‚2.592458114027977 1loss/SRqqt3/detection_loss‚1.1414140677997764 1loss/SRqqt4/detection_loss‚1.4122172690726615RæFaµÛ 0loss/SRqqt2/detection_loss‚2.592458114027977 1loss/SRqqt3/detection_loss‚1.1414140677997764 1loss/SRqqt4/detection_loss‚1.4122172690726615 1loss/FBt1/assignment_loss‚0.32845896511560585 /loss/FBt3/detection_loss‚0.9695379601255285 trainer/global_step‚1749 _runtime‚2071.319556407 _step‚142 /loss/FRt2/assignment_loss‚5.138735751310985 2loss/SRqqt1/assignment_loss‚2.4582864741484323 /loss/FBt4/detection_loss‚3.0600304901599884 epoch‚1 .loss/FBt2/detection_loss‚2.543050597111384 1loss/SRqqt2/assignment_loss‚1.515948401298374 /loss/FRt3/detection_loss‚1.6645975448191166 /loss/FRt4/detection_loss‚1.7459372490262375 1loss/SRqqt1/detection_loss‚2.2301406876908407 % _timestamp‚1.7885266877842543e+09 /loss/FRt1/assignment_loss‚4.746986071268718 0loss/FBt2/assignment_loss‚0.3793385189481569 0loss/FBt4/assignment_loss‚0.2922340748482384 0loss/FRt3/assignment_loss‚1.5473963675590663 0loss/FRt4/assignment_loss‚2.6584985508368564 2loss/SRqqt3/assignment_loss‚1.1491134388923225 /loss/FRt2/detection_loss‚2.7888467450936636 /loss/FBt1/detection_loss‚2.4971117774645486 %loss/total_loss‚47.40253436271518 /loss/FRt1/detection_loss‚2.8553469445970325 2loss/SRqqt4/assignment_loss‚1.3874578049809143 1loss/FBt3/assignment_loss‚0.29939049651875693Ž62çꔶ‚(:‰ í€ëÔù°Œá gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Ì·K·‚(:À í€ëÔ®ÄÒá& proc.memory.availableMB‚ 7.80078125) disk./.usagePercent‚75.90170227222771 disk.dm-0.out‚32586.25390625 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0$ memory_percent‚99.95238780975342 proc.cpu.threads‚17 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 network.recv‚759839 disk.sda.out‚724.65283203125 disk.sda1.out‚0" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273! disk.sda2.out‚724.65283203125# disk.nvme1n1.out‚10640.24609375 disk.rbd0.out‚ 0.62109375" disk.nvme0n1.out‚ 21946.1328125! disk.sdb2.out‚724.65283203125 disk.sda.in‚128.625 disk.sda1.in‚0 disk.rbd2.out‚ 0.783203125 disk.md0.out‚ 715.85546875 network.sent‚ 3.261209e+06 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚724.65283203125 disk.md0p1.out‚ 715.85546875 cpu‚25.165683957913615$ disk./.usageGB‚2714.612522125244 disk.nvme1n1.in‚0 disk.sdb1.in‚0™Ÿ}”¸‚(:‰ ü€ëÔ„ëÓà gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£F¹‚(:» ü€ëÔùûá network.recv‚765841 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.62109375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375$ memory_percent‚99.95238780975342 proc.cpu.threads‚17 disk.sdb2.out‚726.6064453125 network.sent‚ 3.368014e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sda2.out‚726.6064453125 disk.md0.out‚ 717.75390625* proc.memory.percent‚18.190526962280273) disk./.usagePercent‚75.90169469932994 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.sda1.in‚0 disk.sdb.out‚726.6064453125 disk.nvme1n1.out‚ 10683.90625 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.sda.out‚726.6064453125& proc.memory.availableMB‚ 7.80078125" proc.memory.rssMB‚ 2980.3359375% disk./.usageGB‚2714.6122512817383 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚32832.98046875 cpu‚25.164624004793307 disk.md0p1.out‚ 717.75390625# disk.nvme0n1.out‚22149.07421875 disk.sda.in‚128.625WÝÄ”º‚(:‰ ‹ëÔƒ²¢ß gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r ou2»‚(:§ ‹ëÔ¿õìà network.recv‚769468 disk.rbd2.out‚ 0.783203125" disk.nvme0n1.out‚ 22306.4609375 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0* proc.memory.percent‚18.190526962280273 disk.dm-0.out‚ 33063.1328125 disk.sdb.out‚ 728.033203125 cpu‚25.163730697966393 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.out‚ 728.033203125( disk./.usagePercent‚75.9023570612344 disk.rbd2.in‚0! disk.nvme1n1.out‚ 10756.671875 disk.md0p1.out‚719.125" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 3.373372e+06 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sda.out‚ 728.033203125 disk.sda1.out‚0# memory_percent‚99.9547004699707 disk.md0.out‚719.125 disk.dm-0.in‚0$ proc.memory.availableMB‚7.421875$ disk./.usageGB‚2714.635940551758 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚ 728.033203125ürh­2¼- _wandbruntime‚2126  _runtime‚2126k¨Ñ¹Ç½Á trainer/global_step‚1799 % _timestamp‚1.7885267432902884e+09 _runtime‚2126.825743203 _step‚143 & lr-AdamW/pg1‚0.0007265034770467015 & lr-AdamW/pg2‚0.0007265034770467015Ë&ì’Ì¾Æ & lr-AdamW/pg2‚0.0007265034770467015 trainer/global_step‚1799 % _timestamp‚1.7885267432902884e+09 _runtime‚2126.825743203 _step‚143 & lr-AdamW/pg1‚0.0007265034770467015ˆñç2¿- _wandbruntime‚2126  _runtime‚2126WÝrÝî'ÀèO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152407216398336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147472337887381 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885267432912292e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82636967 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2050567971328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _step‚144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22326456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2050567971328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2049658000896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2145122685440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45363187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33193771 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚2126.826841478 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37273780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168117237018624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45363187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2145351468544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37273782 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152406288424960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165972138273280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚147473262901737 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152407216398336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147472337887381 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885267432912292e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82636967 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2050567971328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _step‚144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22326456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2050567971328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2049658000896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2145122685440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45363187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33193771 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚2126.826841478 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37273780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168117237018624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45363187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2145351468544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37273782 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152406288424960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165972138273280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDevic…f)¯ÏeStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚147473262901737 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150438267512832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145509367016160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37293940 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82681958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45414386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150438267512832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2050779461120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145509384055520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150438249509888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45414386 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37293942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33211552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166062363977728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10872391 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚147552472755203 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150438249509888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2146517562880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45388018 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2051689444864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147553397769767 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45388018 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚1799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37293942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2146282501632 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82708328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152489956957696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2044013714247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10873031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144ˆÅP9ý&Ä÷M VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10873031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22338521 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166062340037632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150438267512832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145509367016160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37293940 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82681958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45414386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150438267512832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2050779461120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145509384055520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150438249509888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45414386 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37293942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33211552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166062363977728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10872391 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚147552472755203 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150438249509888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2146517562880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45388018 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2051689444864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147553397769767 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45388018 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚1799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37293942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2146282501632 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82708328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152489956957696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2044013714247‘Ð"jì2Å- _wandbruntime‚2127  _runtime‚2127†®üAWÆÑ /loss/FRt2/detection_loss‚2.2820202189107293 /loss/FRt1/assignment_loss‚4.218214157855872 1loss/SRqqt1/assignment_loss‚2.234158088603327 2loss/SRqqt4/assignment_loss‚1.4755159191598621 /loss/FBt4/detection_loss‚3.1512680053710938 0loss/FRt3/assignment_loss‚1.4459993271600633 /loss/FBt3/detection_loss‚1.0770757866020386 trainer/global_step‚1799 -loss/FRt1/detection_loss‚2.93964440352989 0loss/FRt4/assignment_loss‚3.1049085544868253 /loss/FRt4/detection_loss‚1.8015430707117202 .loss/FBt1/detection_loss‚2.404690880985821 _runtime‚ 2127.93194999 0loss/FBt1/assignment_loss‚0.2403979538105298 _step‚146 /loss/FRt2/assignment_loss‚5.033920795686783 0loss/FBt3/assignment_loss‚0.2325618736975975 % _timestamp‚1.7885267443966398e+09 2loss/SRqqt2/assignment_loss‚1.8307409213767165 1loss/SRqqt3/assignment_loss‚1.197988605957765 /loss/SRqqt2/detection_loss‚2.83353430884225 .loss/FBt2/detection_loss‚2.546960016091665 $loss/total_loss‚46.8484618424563 1loss/FBt2/assignment_loss‚0.20430466863631913 3loss/FBt4/assignment_loss‚0.0038737445138394833 /loss/FRt3/detection_loss‚1.4845388102980832 1loss/SRqqt3/detection_loss‚1.2242836477091679 1loss/SRqqt4/detection_loss‚1.3221020836241637 epoch‚1 1loss/SRqqt1/detection_loss‚2.5582159988341795ã‚kO\ÇÖ /loss/FRt2/assignment_loss‚5.033920795686783 0loss/FBt3/assignment_loss‚0.2325618736975975 % _timestamp‚1.7885267443966398e+09 2loss/SRqqt2/assignment_loss‚1.8307409213767165 1loss/SRqqt3/assignment_loss‚1.197988605957765 /loss/SRqqt2/detection_loss‚2.83353430884225 .loss/FBt2/detection_loss‚2.546960016091665 $loss/total_loss‚46.8484618424563 1loss/FBt2/assignment_loss‚0.20430466863631913 3loss/FBt4/assignment_loss‚0.0038737445138394833 /loss/FRt3/detection_loss‚1.4845388102980832 1loss/SRqqt3/detection_loss‚1.2242836477091679 1loss/SRqqt4/detection_loss‚1.3221020836241637 epoch‚1 1loss/SRqqt1/detection_loss‚2.5582159988341795 /loss/FRt2/detection_loss‚2.2820202189107293 /loss/FRt1/assignment_loss‚4.218214157855872 1loss/SRqqt1/assignment_loss‚2.234158088603327 2loss/SRqqt4/assignment_loss‚1.4755159191598621 /loss/FBt4/detection_loss‚3.1512680053710938 0loss/FRt3/assignment_loss‚1.4459993271600633 /loss/FBt3/detection_loss‚1.0770757866020386 trainer/global_step‚1799 -loss/FRt1/detection_loss‚2.93964440352989 0loss/FRt4/assignment_loss‚3.1049085544868253 /loss/FRt4/detection_loss‚1.8015430707117202 .loss/FBt1/detection_loss‚2.404690880985821 _runtime‚ 2127.93194999 0loss/FBt1/assignment_loss‚0.2403979538105298 _step‚146’òkžÈ‚(:“ šëÔãçòá gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ—œ¿@É‚(:µ šëÔë³®ã disk.sdb1.in‚0 disk.rbd0.in‚0 disk.sda.in‚128.625" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 3.379001e+06 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚729.4013671875 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚729.4013671875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0.in‚ 199.5859375$ proc.memory.availableMB‚7.609375* proc.memory.percent‚18.190526962280273 network.recv‚773280 disk.md0p1.out‚ 720.421875 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375# disk.nvme0n1.out‚22568.51953125 disk.dm-0.in‚0 disk.dm-0.out‚33353.36328125 disk.sdb2.in‚ 70.9609375 disk.sda.out‚729.4013671875 cpu‚25.162877099367954) disk./.usagePercent‚75.90329215411951 disk.sda2.in‚128.625 disk.md0.out‚ 720.421875 disk.sdb.out‚729.4013671875 disk.nvme1n1.out‚ 10784.84375$ memory_percent‚99.95355606079102% disk./.usageGB‚2714.6693840026855lž¨ö’Ê‚(:‡ ©ëÔ˜¸ß gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ópÚAË‚(:¶ ©ëÔ“„ˆá disk.md0p1.in‚ 199.5859375 disk.sdb.in‚ 70.9609375& proc.memory.availableMB‚ 7.35546875* proc.memory.percent‚18.190526962280273 disk.sdb1.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 3.486754e+06) disk./.usagePercent‚75.90942417477173 disk.sdb2.out‚731.2509765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme0n1.out‚ 22568.625 disk.dm-0.out‚33405.51953125 disk.sdb.out‚731.2509765625 proc.cpu.threads‚17 disk.md0p1.out‚ 722.1953125 disk.rbd0.out‚ 0.62109375 disk.sda1.out‚0 disk.sda2.out‚731.2509765625 disk.dm-0.in‚0 network.recv‚780716 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda.out‚731.2509765625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚10836.89453125 disk.nvme0n1.in‚0% disk./.usageGB‚2714.8886947631836 disk.rbd2.in‚0 disk.md0.out‚ 722.1953125 cpu‚25.16192192348534 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375$ memory_percent‚99.95510578155518ŸËx¦“Ì‚(:ˆ ¸ëÔæ´´à gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|Ï*HÍ‚(:½ ¸ëÔú¤„â disk.dm-0.in‚0" proc.memory.rssMB‚ 2980.3359375 cpu‚25.161015309912123 disk.sda1.out‚0 disk.sda2.in‚128.625! disk.sda2.out‚732.41943359375 disk.md0p1.out‚ 723.30078125 disk.sdb2.in‚ 70.9609375 network.sent‚ 3.495127e+06 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125 disk.md0p1.in‚ 199.5859375% disk./.usageGB‚2714.8881607055664 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 22778.2734375& proc.memory.availableMB‚ 7.44921875 disk.sdb.in‚ 70.9609375 disk.sda.out‚732.41943359375 disk.md0.out‚ 723.30078125$ memory_percent‚99.95453357696533* proc.memory.percent‚18.190526962280273 disk.sdb.out‚732.41943359375! disk.sdb2.out‚732.41943359375 disk.nvme1n1.out‚ 10914.53125 disk.rbd0.out‚ 0.62109375 disk.dm-0.out‚ 33692.8046875 proc.cpu.threads‚17 network.recv‚785275) disk./.usagePercent‚75.90940924229723 disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375¾Žê“΂(:ˆ ÇëÔˉßà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.618' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9045 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âSßúKÏ‚(:À ÇëÔÁÞóá disk.dm-0.in‚0 disk.sda.out‚733.88427734375 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚22935.58984375 disk.md0.in‚ 199.5859375" proc.memory.rssMB‚ 2980.3359375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 724.70703125 network.sent‚ 3.498414e+06 disk.sdb2.in‚ 70.9609375% disk./.usageGB‚2715.1391792297363 disk.dm-0.out‚33928.01953125 disk.sdb.out‚733.88427734375 disk.sda.in‚128.625 disk.sda1.in‚0! disk.sdb2.out‚733.88427734375" disk.nvme1n1.out‚ 10992.4296875 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125! disk.sda2.out‚733.88427734375% proc.memory.availableMB‚ 7.8203125 proc.cpu.threads‚17 network.recv‚789368) disk./.usagePercent‚75.91642782529244 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 0.62109375 disk.sda1.out‚0$ memory_percent‚99.95226860046387* proc.memory.percent‚18.190526962280273 disk.md0p1.out‚ 724.70703125 cpu‚25.159920168706474À›¦2Ð- _wandbruntime‚2184  _runtime‚2184 ²äÆÑÀ & lr-AdamW/pg1‚0.0007255704784042212 & lr-AdamW/pg2‚0.0007255704784042212 trainer/global_step‚1849 $ _timestamp‚1.788526801403482e+09 _runtime‚2184.939095922 _step‚147ü /aËÒÅ _runtime‚2184.939095922 _step‚147 & lr-AdamW/pg1‚0.0007255704784042212 & lr-AdamW/pg2‚0.0007255704784042212 trainer/global_step‚1849 $ _timestamp‚1.788526801403482e+09“´ìžì2Ó- _wandbruntime‚2184  _runtime‚2184Õ#ŒÔ:ÔèO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38281780 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2097865720873 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2106655396352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154437602718208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2098773695869 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2204109790720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_byŒŽ£ÿ´ tes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34087573 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2105745425920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46634431 trainer/global_step‚1849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156544258114560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149381229148816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46634431 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149381246188176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885268014049568e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46608089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170483399555584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154437584715264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151479094869689 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46634431 trainer/global_step‚1849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156544258114560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149381229148816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46634431 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149381246188176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885268014049568e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46608089 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170483399555584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154437584715264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151479094869689 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154437602718208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2098773695869 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2204109790720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0”ˆbõ2Ö- _wandbruntime‚2186  _runtime‚2186£ :žø&×òM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149460246743360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34105981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2205069614592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885268025285633e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2186.064620738 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156626998673920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2205285801472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2107776869888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151560154752075ÕjÅ Ø÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154519203801088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46659288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885268025285633e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2186.064620738 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156626998673920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2205285801472 M=DeviceStatsMonitor.on_train_batch_end/reserved·ÎÌñ_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2107776869888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151560154752075 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149460246743360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34105981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2205069614592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155485851851997 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174994708718080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262727515648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 _runtime‚2243.562146686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158518557009408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216200192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11449476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155486776866353 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87169115 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11450103 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚1899 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47879333 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160680371857408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2161832850944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160681299830784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153332108876016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788526860026196e+09 _step‚152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2162742821376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160681299830784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23532956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158518557009408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177257412293632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153332125915376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760{}\éìO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174994684777984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153742975981 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2154650950977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158518539006464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39289782 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23532954 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47879333 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160680371857408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2161832850944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160681299830784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153332108876016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788526860026196e+09 _step‚152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2162742821376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160681299830784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23532956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158518557009408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177257412293632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153332125915376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2162742821376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155485851851997 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174994708718080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 22627275„ÈW– 15648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 _runtime‚2243.562146686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158518557009408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216200192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11449476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155486776866353 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87169115 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11450103 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚1899 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2163864294912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153411143509920 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160764040390144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155768224463 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47877822 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39309942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47877822 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784®N}±ü&ìöM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87187762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87214132 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264116647424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158600158092288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87187762 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39309942 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155565986719819 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23545021 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47904190 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788526861009575e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39309940 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158600176095232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155566911734383 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47904190 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2163864294912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11455890 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2244.546093396 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158600158092288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2263885780480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2154860249259 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11455251 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153411143509920 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160764040390144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155768224463 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47877822 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39309942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47877822 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2163864294912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49124235 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _runtime‚2302.440596294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164818341547008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35877513 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231751680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164817413573632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209620231089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2217920275968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24136204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40297782 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24136206 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181827646510080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚156 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218830246400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11740707 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157282988603216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49097893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181827414758400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2321444758016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89422017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162599493297664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49097893 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164818341547008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207811584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40297782 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2217920275968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210528206085 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49124235 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480fuôó'ýíO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218830246400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40297780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11741307 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89422017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162599493297664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49097893 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164818341547008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207811584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40297782 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2217920275968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210528206085 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49124235 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157283005642576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179505970000384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2321652569600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49124235 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _runtime‚2302.440596294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164818341547008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35877513 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231751680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164817413573632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209620231089 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2217920275968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24136204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40297782 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24136206 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181827646510080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚156 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218830246400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11740707 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157282988603216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49097893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181827414758400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2321444758016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89440664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11747128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210737504367 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49122724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35895399 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2303.487963018 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164901082106368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49149092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2322622008832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159573668716691 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159572743702127 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162681130386432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164900154119680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179596219644928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚1949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24148269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24148271 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 $ _timestamp‚1.788526919951671e+09ƒBÛJW €öM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164900154119680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179596219644928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚1949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24148269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 FDeviceStatsMonitor.on_train_batch_end/reserved_byte@PV¨¥s.all.freed‚0 _step‚157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24148271 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 $ _timestamp‚1.788526919951671e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49149092 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157362006197760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179596195704832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89440664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11747128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210737504367 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49122724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35895399 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2303.487963018 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164901082106368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49149092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2322622008832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159573668716691 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159572743702127 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162681130386432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615ΙDê2- _wandbruntime‚2303  _runtime‚2303‹“û'[‚Õ 0loss/FRt1/assignment_loss‚3.4593426837370944 1loss/SRqqt1/assignment_loss‚1.984493966285999 1loss/SRqqt2/assignment_loss‚2.426633859122241 .loss/FRt2/detection_loss‚3.162037896482568 %loss/total_loss‚49.57029684026314 2loss/SRqqt3/assignment_loss‚1.2515088540586559 1loss/FBt2/assignment_loss‚0.06478919005999412 0loss/FBt4/assignment_loss‚1.2350213000900112 1loss/SRqqt4/detection_loss‚1.3295300006866455 trainer/global_step‚1949 _step‚158 1loss/FBt1/assignment_loss‚0.06843977866685312 /loss/FRt4/detection_loss‚1.9790984854575187 .loss/FBt1/detection_loss‚2.441856801509857 0loss/FRt3/assignment_loss‚1.9519533264009576 1loss/FBt3/assignment_loss‚0.09191686528875555 /loss/FBt3/detection_loss‚0.6741692202568457 /loss/FBt4/detection_loss‚2.4018249809741974 % _timestamp‚1.7885269199541929e+09 _runtime‚2303.489463485 epoch‚1 /loss/FRt3/detection_loss‚1.8614444075511736 1loss/SRqqt1/detection_loss‚2.9300931233626146 1loss/SRqqt3/detection_loss‚1.5207976257259195 /loss/FRt1/detection_loss‚3.2236602833637824 /loss/FBt2/detection_loss‚3.2921153936121197 /loss/FRt2/assignment_loss‚4.592800918378328 2loss/SRqqt4/assignment_loss‚1.4457385351444467 1loss/SRqqt2/detection_loss‚2.6865410650217973 .loss/FRt4/assignment_loss‚3.49448827902476´å¹`ƒÚ /loss/FRt4/detection_loss‚1.9790984854575187 .loss/FBt1/detection_loss‚2.441856801509857 0loss/FRt3/assignment_loss‚1.9519533264009576 1loss/FBt3/assignment_loss‚0.09191686528875555 /loss/FBt3/detection_loss‚0.6741692202568457 /loss/FBt4/detection_loss‚2.4018249809741974 % _timestamp‚1.7885269199541929e+09 _runtime‚2303.489463485 epoch‚1 /loss/FRt3/detection_loss‚1.8614444075511736 1loss/SRqqt1/detection_loss‚2.9300931233626146 1loss/SRqqt3/detection_loss‚1.5207976257259195 /loss/FRt1/detection_loss‚3.2236602833637824 /loss/FBt2/detection_loss‚3.2921153936121197 /loss/FRt2/assignment_loss‚4.592800918378328 2loss/SRqqt4/assignment_loss‚1.4457385351444467 1loss/SRqqt2/detection_loss‚2.6865410650217973 .loss/FRt4/assignment_loss‚3.49448827902476 0loss/FRt1/assignment_loss‚3.4593426837370944 1loss/SRqqt1/assignment_loss‚1.984493966285999 1loss/SRqqt2/assignment_loss‚2.426633859122241 .loss/FRt2/detection_loss‚3.162037896482568 %loss/total_loss‚49.57029684026314 2loss/SRqqt3/assignment_loss‚1.2515088540586559 1loss/FBt2/assignment_loss‚0.06478919005999412 0loss/FBt4/assignment_loss‚1.2350213000900112 1loss/SRqqt4/detection_loss‚1.3295300006866455 trainer/global_step‚1949 _step‚158 1loss/FBt1/assignment_loss‚0.06843977866685312ž1Ö©®”„‚(:‰ ΂ëÔ›Ú‰á gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M3®E…‚(:º ΂ëÔÆå¾á disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 737.41015625 disk.sdb1.out‚0 disk.nvme0n1.out‚24354.25 disk.sdb.out‚747.12841796875& proc.memory.availableMB‚ 7.67578125) disk./.usagePercent‚75.91640126681995 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚36087.47265625 disk.rbd2.in‚0 disk.md0.out‚ 737.41015625 disk.dm-0.in‚0 disk.sda.out‚747.12841796875 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 network.sent‚ 3.754139e+06$ disk./.usageGB‚2715.138229370117$ memory_percent‚99.95315074920654 cpu‚25.15100041610406 proc.cpu.threads‚17 network.recv‚831723 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚747.12841796875# disk.nvme1n1.out‚11733.22265625 disk.rbd0.out‚ 0.62109375! disk.sda2.out‚747.12841796875 disk.sdb.in‚ 70.9609375˜Ñ]&“†‚(:ˆ Ý‚ëÔ™·à gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Wj9C‡‚(:¸ Ý‚ëÔÁ²Ðà network.sent‚ 3.861937e+06 network.recv‚838668" disk.nvme0n1.out‚ 24455.8515625 disk.sdb.in‚ 70.9609375 disk.sda.out‚748.53857421875 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚36347.75$ memory_percent‚99.95310306549072 disk.sda1.out‚0 disk.rbd2.out‚ 0.783203125 disk.md0.out‚ 738.76171875" disk.nvme1n1.out‚ 11891.8984375 disk.rbd0.in‚0& proc.memory.availableMB‚ 7.68359375 cpu‚25.15016194500597 disk.rbd2.in‚0 disk.sda2.in‚128.625) disk./.usagePercent‚75.91640126681995$ disk./.usageGB‚2715.138229370117! disk.sda2.out‚748.53857421875 disk.sda.in‚128.625 disk.md0p1.out‚ 738.76171875! disk.sdb2.out‚748.53857421875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚ 0.62109375" proc.memory.rssMB‚ 2980.3359375 disk.sdb.out‚748.53857421875 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚18.190526962280273™"ÍO’ˆ‚(:‡ ì‚ëÔß´–à gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kŸù«G‰‚(:¼ ì‚ëÔÆÉà disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0" proc.memory.rssMB‚ 2980.3359375# disk.nvme0n1.out‚24665.69921875 disk.sda2.out‚749.9814453125 disk.sda.out‚749.9814453125% proc.memory.availableMB‚ 7.6796875 network.sent‚ 3.870202e+06 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 11969.7109375 disk.sdb1.out‚0$ memory_percent‚99.95312690734863* proc.memory.percent‚18.190526962280273 disk.md0p1.out‚ 740.14453125 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375) disk./.usagePercent‚75.90046724992628 disk.rbd0.out‚ 0.62109375 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚128.625 disk.md0.out‚ 740.14453125 disk.sdb2.out‚749.9814453125 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 network.recv‚842896% disk./.usageGB‚2714.5683517456055 disk.sda.in‚128.625 cpu‚25.149527496488975 disk.md0p1.in‚ 199.5859375 disk.md0.in‚ 199.5859375 disk.dm-0.out‚36635.41015625 disk.sdb.out‚749.9814453125 proc.cpu.threads‚17Âìb©”Š‚(:‰ û‚ëÔ‚¾–à gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÑ.;‹‚(:° û‚ëÔàåÊà cpu‚25.148794687028527! disk.sdb2.out‚751.64990234375 disk.sda.out‚751.64990234375 disk.nvme1n1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.783203125 disk.sda2.in‚128.625 disk.sdb.out‚751.64990234375 disk.sdb2.in‚ 70.9609375 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚741.75* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 network.sent‚ 3.875592e+06) disk./.usagePercent‚75.90165022188803 disk.sda1.in‚0 disk.nvme0n1.out‚ 24822.9375 disk.dm-0.out‚36836.33203125 network.recv‚846601 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0% disk./.usageGB‚2714.6106605529785 disk.dm-0.in‚0 disk.sda.in‚128.625# disk.nvme1n1.out‚12013.39453125 disk.sdb1.in‚0$ proc.memory.availableMB‚7.671875 disk.sdb1.out‚0! disk.sda2.out‚751.64990234375 disk.md0.out‚741.75 disk.rbd0.out‚ 0.62109375$ memory_percent‚99.95317459106445" proc.memory.rssMB‚ 2980.3359375¬Ø¯2Œ- _wandbruntime‚2361  _runtime‚2361¸õ‘¸ÇÁ & lr-AdamW/pg1‚0.0007223226092175194 & lr-AdamW/pg2‚0.0007223226092175194 trainer/global_step‚1999 % _timestamp‚1.7885269776799643e+09 _runtime‚2361.215466169 _step‚159†ë÷ ÌŽÆ & lr-AdamW/pg1‚0.0007223226092175194 & lr-AdamW/pg2‚0.0007223226092175194 trainer/global_step‚1999 % _timestamp‚1.7885269776799643e+09 _runtime‚2361.215466169 _step‚159Ÿد6Z2- _wandbruntime‚2361  _runtime‚2361 ‡sÎî'èO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _step‚160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2380180726784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36771786 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2274917671424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166680447588864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41305780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161233868330416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚1999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91674919 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41305782 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163500290830969 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50369137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166680447588864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184017255222784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41305780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166680465591808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50369137 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2274917671424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2274007700992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265497486197 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166680465591808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186397435949568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41305782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12032330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161233885369776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2380405315584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ò´‘íO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265497486197 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166680465591808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186397435949568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41305782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12032330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161233885369776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2380405315584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _step‚160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2380180726784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 eQDeviceStats/Fê?Monitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36771786 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2274917671424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166680447588864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41305780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161233868330416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚1999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91674919 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41305782 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163500290830969 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50369137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166680447588864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184017255222784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41305780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166680465591808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50369137 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2274917671424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2274007700992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 W›eŠ2’- _wandbruntime‚2362  _runtime‚2362îÓÛG÷&“ñM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50367626 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50393994 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚ 2362.38043402 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24751519 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚161 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41325942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184107504867328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161312902964320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885269788441572e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50367626 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2275129161216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186488843713536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570<'î¿ü&”öM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2275129161216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186488843713536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50367626 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50393994 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚ 2362.38043402 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24751519 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚161 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41325942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184107504867328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161312902964320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885269788441572e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50367626 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128¡¬Z2•- _wandbruntime‚2362  _runtime‚2362¨º=h[–Õ 1loss/FBt3/assignment_loss‚0.13661871825981642 _step‚162 2loss/SRqqt2/assignment_loss‚1.7083894621048654 0loss/FBt4/assignment_loss‚0.4378283209633082 .loss/FRt1/detection_loss‚2.855020336310069 %loss/total_loss‚47.65988389630469 _runtime‚2362.381466083 1loss/FBt1/assignment_loss‚0.12040368228915416 1loss/SRqqt1/detection_loss‚2.6417101210561293 1loss/SRqqt3/detection_loss‚1.2680489906952495 epoch‚1 /loss/FRt1/assignment_loss‚3.689550773302714 0loss/FRt3/assignment_loss‚2.0051611959934235 0loss/SRqqt2/detection_loss‚2.571911918265479 1loss/SRqqt4/detection_loss‚1.4655562136854445 /loss/FBt1/detection_loss‚3.2044708828131356 $ _timestamp‚1.788526978846174e+09 0loss/FRt4/assignment_loss‚2.9835178096379553 /loss/FBt3/detection_loss‚0.8424927636806387 /loss/FBt4/detection_loss‚2.4604745268821717 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚5.375167392194271 2loss/SRqqt1/assignment_loss‚2.5161102064724625 /loss/FRt2/detection_loss‚2.3089168816804886 /loss/FRt4/detection_loss‚2.1202999391992177 2loss/SRqqt4/assignment_loss‚1.4151576776589667 /loss/FBt2/detection_loss‚2.4107301334540048 /loss/FRt3/detection_loss‚1.6013906354983818 2loss/SRqqt3/assignment_loss‚1.2595241684170941 /loss/FBt2/assignment_loss‚0.2614311457902458–-ô`—Ú /loss/FRt2/assignment_loss‚5.375167392194271 2loss/SRqqt1/assignment_loss‚2.5161102064724625 /loss/FRt2/detection_loss‚2.3089168816804886 /loss/FRt4/detection_loss‚2.1202999391992177 2loss/SRqqt4/assignment_loss‚1.4151576776589667 /loss/FBt2/detection_loss‚2.4107301334540048 /loss/FRt3/detection_loss‚1.6013906354983818 2loss/SRqqt3/assignment_loss‚1.2595241684170941 /loss/FBt2/assignment_loss‚0.261431145790245 1loss/FBt3/assignment_loss‚0.13661871825981642 _step‚162 2loss/SRqqt2/assignment_loss‚1.7083894621048654 0loss/FBt4/assignment_loss‚0.4378283209633082 .loss/FRt1/detection_loss‚2.855020336310069 %loss/total_loss‚47.65988389630469 _runtime‚2362.381466083 1loss/FBt1/assignment_loss‚0.12040368228915416 1loss/SRqqt1/detection_loss‚2.6417101210561293 1loss/SRqqt3/detection_loss‚1.2680489906952495 epoch‚1 /loss/FRt1/assignment_loss‚3.689550773302714 0loss/FRt3/assignment_loss‚2.0051611959934235 0loss/SRqqt2/detection_loss‚2.571911918265479 1loss/SRqqt4/detection_loss‚1.4655562136854445 /loss/FBt1/detection_loss‚3.2044708828131356 $ _timestamp‚1.788526978846174e+09 0loss/FRt4/assignment_loss‚2.9835178096379553 /loss/FBt3/detection_loss‚0.8424927636806387 /loss/FBt4/detection_loss‚2.4604745268821717 trainer/global_step‚1999¢;©Q]’˜‚(:‡ ŠƒëÔн’á gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽC›aH™‚(:½ ŠƒëÔðÌá network.recv‚850647) disk./.usagePercent‚75.90237743339604 disk.rbd0.out‚ 0.62109375$ memory_percent‚99.95460510253906* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 disk.sda.out‚752.94580078125 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 network.sent‚ 3.879155e+06 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375! disk.sdb2.out‚752.94580078125 disk.md0p1.out‚ 742.98828125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.in‚0 disk.dm-0.out‚37135.88671875" proc.memory.availableMB‚7.4375 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625 disk.md0.out‚ 742.98828125% disk./.usageGB‚2714.6366691589355 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚12078.33203125 disk.sda1.in‚0" disk.nvme0n1.out‚ 25085.4296875! disk.sda2.out‚752.94580078125 disk.dm-0.in‚0" proc.memory.rssMB‚ 2980.3359375 disk.sdb.out‚752.94580078125 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 cpu‚25.148048852434947:i˜=”š‚(:‰ ™ƒëÔÑ­µà gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g/ì†2›‚(:§ ™ƒëÔä‰Ñà) disk./.usagePercent‚75.90331252628116 disk.sda.in‚128.625# disk.nvme1n1.out‚12088.66796875 network.recv‚856955 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 0.62109375 disk.rbd2.out‚ 0.783203125 disk.md0.in‚ 199.5859375 disk.dm-0.out‚37331.34765625" proc.memory.availableMB‚7.6875" proc.memory.rssMB‚ 2980.3359375 disk.sdb2.in‚ 70.9609375 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.sda2.out‚ 754.3515625 disk.md0.out‚ 744.34375$ memory_percent‚99.95307922363281 network.sent‚ 3.986197e+06 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 25242.6796875 cpu‚25.147356039447953% disk./.usageGB‚2714.6701126098633 disk.sdb.out‚ 754.3515625 proc.cpu.threads‚17 disk.sdb2.out‚ 754.3515625 disk.md0p1.out‚ 744.34375* proc.memory.percent‚18.190526962280273 disk.sda.out‚ 754.3515625 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375x󄆓œ‚(:ˆ ¨ƒëÔþÔà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡øò$:‚(:¯ ¨ƒëÔª”á disk.dm-0.out‚ 37565.328125$ memory_percent‚99.95307922363281" proc.memory.availableMB‚7.6875 network.sent‚ 3.992342e+06 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚756.2587890625! disk.nvme1n1.out‚ 12166.359375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 746.1796875 proc.cpu.threads‚17 network.recv‚861053 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚756.2587890625 disk.sda.in‚128.625 disk.rbd2.out‚ 0.783203125 disk.sda2.out‚756.2587890625% disk./.usageGB‚2714.8892364501953 disk.sda.out‚756.2587890625 disk.nvme1n1.in‚0 cpu‚25.14663746366951 disk.md0p1.out‚ 746.1796875 disk.rbd0.out‚ 0.62109375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0( disk./.usagePercent‚75.9094393205673 disk.nvme0n1.out‚ 25398.96875" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 disk.sda1.out‚0 disk.dm-0.in‚0£^—”ž‚(:‰ ·ƒëÔôÙÅà gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ı;6EŸ‚(:¹ ·ƒëԟ׸á" disk.nvme1n1.out‚ 12243.9921875 disk.rbd2.out‚ 0.794921875 network.sent‚ 3.99983e+06 network.recv‚865078 disk.dm-0.out‚37696.56640625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 747.5859375 disk.rbd0.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 747.5859375" proc.memory.availableMB‚7.4375" proc.memory.rssMB‚ 2980.3359375% disk./.usageGB‚2714.8887214660645 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚757.71826171875) disk./.usagePercent‚75.90942492139546# diððÖÿsk.nvme0n1.out‚25455.19921875 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.190526962280273 disk.rbd0.out‚ 0.69921875! disk.sda2.out‚757.71826171875 disk.dm-0.in‚0 disk.sda.out‚757.71826171875 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625$ memory_percent‚99.95460510253906 proc.cpu.threads‚17 disk.sdb.out‚757.71826171875 disk.sdb1.in‚0 disk.sda1.out‚0 cpu‚25.14593588912898 disk.sda.in‚128.625\Û…Þ2 - _wandbruntime‚2419  _runtime‚2419O!RÇ¡Á & lr-AdamW/pg1‚0.0007210911137975014 & lr-AdamW/pg2‚0.0007210911137975014 trainer/global_step‚2049 % _timestamp‚1.7885270363151824e+09 _runtime‚2419.850808302 _step‚163z« qÌ¢Æ _runtime‚2419.850808302 _step‚163 & lr-AdamW/pg1‚0.0007210911137975014 & lr-AdamW/pg2‚0.0007210911137975014 trainer/global_step‚2049 % _timestamp‚1.7885270363151824e+09£(¬”2£- _wandbruntime‚2419  _runtime‚2419‰>ë›î'¤èO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173091497006080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188528564385280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167506122798921 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37665714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2330095126016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170761419883008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25342704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚164 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42313780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170761401880064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93927821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2321374741305 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439104470016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51587697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12323008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170761401880064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2439316475904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167507047813277 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37665087 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 _runtime‚2419.852840431 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¯ ´Äó'¥íO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170761401880064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2439316475904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167507047813277 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37665087 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 _runtime‚2419.852840431 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173091497006080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188528564385280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167506122798921 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37665714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2330095126016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170761419883008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25342704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚164 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42313780 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170761401880064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93927821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2321374741305 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439104470016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51587697 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12323008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¤º„ˆ2¦- _wandbruntime‚2421  _runtime‚2421wt ø&§òM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188618790089728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12328845 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332126569984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173175165538816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332126569984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165263782691520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440264051712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885270374977415e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51612528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25354769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚2421.033944645 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170843020965888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12328198 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165263765652160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2331216586240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167587182681307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25354771 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440497015808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2331216586240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167587182681307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.largúq2BD"e_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25354771 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440497015808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332126569984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173175165538816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332126569984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165263782691520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440264051712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885270374977415e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51612528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25354769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚2421.033944645 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170843020965888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12328198 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165263765652160 E%¦^ªØ /loss/FRt4/assignment_loss‚2.866685607309999 .loss/FBt2/detection_loss‚2.023069759209951 /loss/FBt4/detection_loss‚3.1175559639930723 _runtime‚2421.035262271 0loss/FRt3/assignment_loss‚1.7886861543501578 2loss/SRqqt2/assignment_loss‚1.8992028757929802 /loss/FRt3/detection_loss‚1.6359576707886112 1loss/SRqqt4/detection_loss‚1.3269667866421335 epoch‚1 2loss/SRqqt1/assignment_loss‚2.1917883813381196 trainer/global_step‚2049 _step‚166 /loss/FRt2/assignment_loss‚4.673123317105429 0loss/FBt1/assignment_loss‚0.4362271978429312 /loss/FBt1/detection_loss‚2.6220078468322754 2loss/SRqqt3/assignment_loss‚1.0653409156948328 /loss/FRt1/detection_loss‚3.3730999556454746 1loss/SRqqt3/detection_loss‚0.9269267751951702 /loss/FBt3/detection_loss‚0.5158752923934824 1loss/FBt4/assignment_loss‚0.08197732499102131 .loss/FRt4/detection_loss‚2.282262080702288 1loss/SRqqt1/detection_loss‚3.1220141515135764 % _timestamp‚1.7885270374998868e+09 0loss/FRt1/assignment_loss‚4.3191796676679095 2loss/SRqqt4/assignment_loss‚1.3248385379987735 1loss/FBt2/assignment_loss‚0.34428288352986175 1loss/SRqqt2/detection_loss‚2.7320270985364914 %loss/total_loss‚47.28428797116802 1loss/FBt3/assignment_loss‚0.13309945413539784 /loss/FRt2/detection_loss‚2.4820922719580785QTWúc«Ý /loss/FRt2/assignment_loss‚4.673123317105429 0loss/FBt1/assignment_loss‚0.4362271978429312 /loss/FBt1/detection_loss‚2.6220078468322754 2loss/SRqqt3/assignment_loss‚1.0653409156948328 /loss/FRt1/detection_loss‚3.3730999556454746 1loss/SRqqt3/detection_loss‚0.9269267751951702 /loss/FBt3/detection_loss‚0.5158752923934824 1loss/FBt4/assignment_loss‚0.08197732499102131 .loss/FRt4/detection_loss‚2.282262080702288 1loss/SRqqt1/detection_loss‚3.1220141515135764 % _timestamp‚1.7885270374998868e+09 0loss/FRt1/assignment_loss‚4.3191796676679095 2loss/SRqqt4/assignment_loss‚1.3248385379987735 1loss/FBt2/assignment_loss‚0.34428288352986175 1loss/SRqqt2/detection_loss‚2.7320270985364914 %loss/total_loss‚47.28428797116802 1loss/FBt3/assignment_loss‚0.13309945413539784 /loss/FRt2/detection_loss‚2.4820922719580785 /loss/FRt4/assignment_loss‚2.866685607309999 .loss/FBt2/detection_loss‚2.023069759209951 /loss/FBt4/detection_loss‚3.1175559639930723 _runtime‚2421.035262271 0loss/FRt3/assignment_loss‚1.7886861543501578 2loss/SRqqt2/assignment_loss‚1.8992028757929802 /loss/FRt3/detection_loss‚1.6359576707886112 1loss/SRqqt4/detection_loss‚1.3269667866421335 epoch‚1 2loss/SRqqt1/assignment_loss‚2.1917883813381196 trainer/global_step‚2049 _step‚166¦1ô!•“¬‚(:ˆ ƃëÔþÂÎá gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>WYE­‚(:º ƃëÔ¨­íâ disk.sda2.in‚128.625 network.sent‚ 4.003334e+06) disk./.usagePercent‚75.91643241169533# disk.nvme1n1.out‚12322.07421875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.69921875" disk.nvme0n1.out‚ 25610.0234375 network.recv‚869235 disk.sdb.in‚ 70.9609375 disk.sdb.out‚758.83642578125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 748.65625 disk.nvme1n1.in‚0% proc.memory.availableMB‚ 7.6484375 disk.md0p1.out‚ 748.65625 disk.md0.in‚ 199.5859375$ memory_percent‚99.95331764221191* proc.memory.percent‚18.190526962280273 disk.dm-0.out‚37932.09765625 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 cpu‚25.145319120806032 disk.sda.out‚758.83642578125 disk.sdb1.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2715.1393432617188" proc.memory.rssMB‚ 2980.3359375 disk.dm-0.in‚0! disk.sdb2.out‚758.83642578125 disk.sda.in‚128.625 disk.sda1.in‚0 disk.rbd2.out‚ 0.857421875! disk.sda2.out‚758.83642578125 disk.nvme0n1.in‚0ò?8N”®‚(:‰ ÕƒëÔ–Ç´à gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—%@+6¯‚(:« ÕƒëÔéšµá disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0# disk.nvme0n1.out‚25819.76171875 disk.dm-0.out‚38296.93359375 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 760.48046875 cpu‚25.144568420460985 proc.cpu.threads‚17% disk./.usageGB‚2715.1383476257324 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.91640457329643 disk.sda1.out‚0 disk.sda.out‚ 760.48046875 disk.nvme1n1.in‚0$ memory_percent‚99.95334148406982 disk.rbd0.out‚ 0.78662109375 disk.sdb1.in‚0 disk.rbd2.out‚ 0.857421875! disk.nvme1n1.out‚ 12477.171875 disk.nvme0n1.in‚0 disk.md0.out‚750.25 disk.dm-0.in‚0 disk.sdb.out‚ 760.48046875& proc.memory.availableMB‚ 7.64453125 network.sent‚ 4.113175e+06 disk.sda1.in‚0 disk.sda2.out‚ 760.48046875 disk.sdb2.in‚ 70.9609375 network.recv‚876564 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.md0p1.out‚750.25* proc.memory.percent‚18.190526962280273‚zä“°‚(:ˆ äƒëÔç‹á gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I^†M±‚(: äƒëÔó×á$ proc.memory.availableMB‚7.140625* proc.memory.percent‚18.190550804138184 cpu‚25.14389087728339$ memory_percent‚99.95641708374023 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0# proc.memory.rssMB‚ 2980.33984375 disk.sdb.out‚761.92041015625) disk./.usagePercent‚75.90166056795965% disk./.usageGB‚2714.6110305786133 disk.nvme0n1.in‚0# disk.nvme0n1.out‚26029.71484375! disk.sda2.out‚761.92041015625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 751.62890625 network.sent‚ 4.11716e+06 disk.dm-0.out‚38595.58203125 disk.sda.out‚761.92041015625" disk.nvme1n1.out‚ 12565.8671875 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 751.62890625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.78662109375 disk.sdb1.out‚0 network.recv‚881094! disk.sdb2.out‚761.92041015625 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.sda.in‚128.625 disk.rbd2.out‚ 0.9365234375Τ2´- _wandbruntime‚2478  _runtime‚2478ûÏfǵÁ trainer/global_step‚2099 % _timestamp‚1.7885270947887306e+09 _runtime‚2478.324246258 _step‚167 & lr-AdamW/pg1‚0.0007197855970270293 & lr-AdamW/pg2‚0.0007197855970270293¿ÏüµÌ¶Æ _step‚167 & lr-AdamW/pg1‚0.0007197855970270293 & lr-AdamW/pg2‚0.0007197855970270293 trainer/global_step‚2099 % _timestamp‚1.7885270947887306e+09 _runtime‚2478.324246258§JoWî2·- _wandbruntime‚2478  _runtime‚2478S’…\î'¸èO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193039849607680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193039825667584 _runtime‚2478.325355793 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 trainer/global_step‚2099 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12614376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43321782 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52832599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169135644824176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52858941 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559713 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25945954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498071551488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174842374174208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174842356171264 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2378159971409 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43321780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386182551040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25945956 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168¦‰øð¹íO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 trainer/global_step‚2099 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12614376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43321782 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52832599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169135644824176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52858941 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559713 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25945954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498071551488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174842374174208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174842356171264 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2378159971409 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43321780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386182551040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25945956 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193039849607680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193039825667584 _runtime‚2478.325355793 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570¨{‚2º- _wandbruntime‚2479  _runtime‚24797T÷÷&»ñM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96225740 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52883798 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193130075312128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378369269691 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2387304011264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2498997241344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚2479.399358169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2379277244895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25958021 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177312207255040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52857430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12620145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38578166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43341940 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96199370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174923993260032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788527095862963e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2388213995008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193130051372032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _step‚169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174923975257088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52883798 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2499238594048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25958019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Å–÷¢ü&¼öM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38578166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43341940 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96199370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174923993260032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788527095862963e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2388213995008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193130051372032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _step‚169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174923975257088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52883798 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2499238594048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25958019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96225740 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52883798 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193130075312128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378369269691 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2387304011264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2498997241344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚2479.399358169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2379277244895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25958021 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177312207255040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52857430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12620145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368©ãQ˜2½- _wandbruntime‚2479  _runtime‚2479ïóX¾Ò 2loss/SRqqt4/assignment_loss‚1.3988980777561664 /loss/FRt1/detection_loss‚2.9656941931823204 1loss/SRqqt1/detection_loss‚2.9607519298791884 _runtime‚2479.400762655 0loss/FRt1/assignment_loss‚3.6261554380943037 /loss/FRt4/assignment_loss‚3.172555791797923 1loss/FBt3/assignment_loss‚0.16130098938898577 $loss/total_loss‚47.6120902413313 /loss/FRt3/detection_loss‚1.8984137112274766 .loss/FRt4/detection_loss‚1.879444942879143 /loss/FBt1/detection_loss‚1.7762569071991103 /loss/FBt3/detection_loss‚0.8109367717085267 .loss/FBt4/detection_loss‚3.373758006095886 epoch‚1 /loss/FRt2/assignment_loss‚5.055892467498779 1loss/FBt1/assignment_loss‚0.46963906441147024 /loss/FBt4/assignment_loss‚0.468095875531435 0loss/SRqqt2/detection_loss‚2.539557392398516 _step‚170 0loss/FRt3/assignment_loss‚1.9231609712753976 2loss/SRqqt1/assignment_loss‚1.6812449783086776 1loss/SRqqt4/detection_loss‚1.3961291346166815 /loss/FBt2/detection_loss‚2.2242213249206544 trainer/global_step‚2099 .loss/FRt2/detection_loss‚3.107991945743561 1loss/SRqqt2/assignment_loss‚2.055987765391668 2loss/SRqqt3/assignment_loss‚1.0983963162982708 0loss/FBt2/assignment_loss‚0.4161921362056243 1loss/SRqqt3/detection_loss‚1.1514141095215327 % _timestamp‚1.7885270958654015e+09¢Ê-‰]¿× .loss/FRt2/detection_loss‚3.107991945743561 1loss/SRqqt2/assignment_loss‚2.055987765391668 2loss/SRqqt3/assignment_loss‚1.0983963162982708 0loss/FBt2/assignment_loss‚0.4161921362056243 1loss/SRqqt3/detection_loss‚1.1514141095215327 % _timestamp‚1.7885270958654015e+09 2loss/SRqqt4/assignment_loss‚1.3988980777561664 /loss/FRt1/detection_loss‚2.9656941931823204 1loss/SRqqt1/detection_loss‚2.9607519298791884 _runtime‚2479.400762655 0loss/FRt1/assignment_loss‚3.6261554380943037 /loss/FRt4/assignment_loss‚3.172555791797923 1loss/FBt3/assignment_loss‚0.16130098938898577 $loss/total_loss‚47.6120902413313 /loss/FRt3/detection_loss‚1.8984137112274766 .loss/FRt4/detection_loss‚1.879444942879143 /loss/FBt1/detection_loss‚1.7762569071991103 /loss/FBt3/detection_loss‚0.8109367717085267 .loss/FBt4/detection_loss‚3.373758006095886 epoch‚1 /loss/FRt2/assignment_loss‚5.055892467498779 1loss/FBt1/assignment_loss‚0.46963906441147024 /loss/FBt4/assignment_loss‚0.468095875531435 0loss/SRqqt2/detection_loss‚2.539557392398516 _step‚170 0loss/FRt3/assignment_loss‚1.9231609712753976 2loss/SRqqt1/assignment_loss‚1.6812449783086776 1loss/SRqqt4/detection_loss‚1.3961291346166815 /loss/FBt2/detection_loss‚2.2242213249206544 trainer/global_step‚2099ªœ¥ˆÇ‘À‚(:† ‚„ëÔ¬ÕÜà gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Œ[(5Á‚(:ª ‚„ëÔ…‹¿â( disk./.usagePercent‚75.9023782866803 disk.md0.out‚ 754.03125 disk.sdb.in‚ 70.9609375 disk.sda.out‚ 764.4140625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 754.03125 disk.rbd0.in‚0$ memory_percent‚99.95415210723877 network.recv‚889349 disk.rbd2.out‚ 0.99951171875# disk.nvme0n1.out‚26397.23046875 disk.sdb2.in‚ 70.9609375& proc.memory.availableMB‚ 7.51171875 network.sent‚ 4.141116e+06 disk.md0.in‚ 199.5859375" disk.nvme1n1.out‚ 12681.6796875 disk.rbd0.out‚ 0.849609375% disk./.usageGB‚2714.6366996765137 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 proc.memory.rssMB‚ 2980.34375* proc.memory.percent‚18.190574645996094 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.dm-0.out‚39078.91015625 disk.sda.in‚128.625 disk.nvme1n1.in‚0 cpu‚25.14255451818449 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.sda2.out‚ 764.4140625 disk.dm-0.in‚0 disk.rbd2.in‚0 disk.sdb.out‚ 764.4140625 disk.sdb2.out‚ 764.4140625/0Œ”‚(:‰ ‘„ëÔžÝâà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°&&‡FÂ(:» ‘„ëÔÉùÝá disk.dm-0.in‚0 disk.sda.out‚766.13818359375$ memory_percent‚99.95417594909668 cpu‚25.141922557758733 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 39107.046875 disk.sdb2.in‚ 70.9609375 network.recv‚897073% proc.memory.availableMB‚ 7.5078125 network.sent‚ 4.238136e+06 disk.nvme1n1.out‚ 12709.8125 disk.rbd2.out‚ 0.99951171875 disk.nvme0n1.in‚0 disk.sdb.out‚766.13818359375 proc.cpu.threads‚17$ disk./.usageGB‚2714.670024871826 disk.sdb1.out‚0* proc.memory.percent‚18.190574645996094) disk./.usagePercent‚75.90331007308892 disk.md0p1.out‚ 755.69921875 disk.rbd0.out‚ 0.91259765625 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.md0.out‚ 755.69921875 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚766.13818359375 proc.memory.rssMB‚ 2980.34375 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0! disk.sda2.out‚766.13818359375 disk.nvme1n1.in‚0 disk.sdb1.in‚0! disk.nvme0n1.out‚ 26397.234375 disk.sda2.in‚128.625 þ”Ä‚(:‰  „ëÔÜõñà gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¬ وł(:·  „ëÔ°˜¸á disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.91259765625$ memory_percent‚99.95577335357666 cpu‚25.141288388498552 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚1.0625 disk.md0.out‚ 757.296875 disk.sdb.in‚ 70.9609375 proc.memory.rssMB‚ 2980.34375* proc.memory.percent‚18.190574645996094 network.sent‚ 4.244351e+06 network.recv‚901859( disk./.usagePercent‚75.9094260946613 disk.sda.in‚128.625 disk.sda.out‚767.79345703125 disk.md0p1.out‚ 757.296875 disk.sdb1.in‚0! disk.sda2.out‚767.79345703125 disk.NŒìòºnvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.sdb.out‚767.79345703125& proc.memory.availableMB‚ 7.24609375 proc.cpu.threads‚17% disk./.usageGB‚2714.8887634277344# disk.nvme1n1.out‚12813.47265625 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚39420.51953125! disk.nvme0n1.out‚ 26607.046875 disk.sda2.in‚128.625! disk.sdb2.out‚767.79345703125 disk.sda1.in‚0 disk.md0.in‚ 199.5859375‚´­¡”Æ‚(:‰ ¯„ëÔŒäÀà gpu.0.gpu‚15.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09‘7‹CÇ‚(:¸ ¯„ëÔãÉíá* proc.memory.percent‚18.190574645996094 disk.sda.in‚128.625! disk.sda2.out‚768.93603515625 network.sent‚ 4.249732e+06 disk.md0p1.out‚ 758.3828125 cpu‚25.140647039071037) disk./.usagePercent‚75.91644467765651! disk.sdb2.out‚768.93603515625 disk.rbd2.in‚0 disk.dm-0.in‚0 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.91259765625 disk.sda1.out‚0 disk.md0.in‚ 199.5859375$ memory_percent‚99.95594024658203# proc.memory.availableMB‚7.21875 disk.sdb2.in‚ 70.9609375 disk.sda.out‚768.93603515625" disk.nvme1n1.out‚ 12909.7109375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0% disk./.usageGB‚2715.1397819519043 disk.sdb.out‚768.93603515625 disk.rbd2.out‚1.0625# disk.nvme0n1.out‚26764.28515625 disk.dm-0.out‚39673.99609375 disk.sdb1.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2980.34375 disk.md0.out‚ 758.3828125 network.recv‚905887 disk.nvme0n1.in‚0 disk.sda2.in‚128.625¦xX¤2È- _wandbruntime‚2537  _runtime‚25374ØLAÇÉÁ & lr-AdamW/pg1‚0.0007184063309738414 & lr-AdamW/pg2‚0.0007184063309738414 trainer/global_step‚2149 % _timestamp‚1.7885271536432135e+09 _runtime‚2537.178686733 _step‚171–μÔÌÊÆ & lr-AdamW/pg1‚0.0007184063309738414 & lr-AdamW/pg2‚0.0007184063309738414 trainer/global_step‚2149 % _timestamp‚1.7885271536432135e+09 _runtime‚2537.178686733 _step‚171«ÒÁî2Ë- _wandbruntime‚2537  _runtime‚2537»G í'ÌçO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚2149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178923310462464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2434037226517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2443179946496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54103843 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 $ _timestamp‚1.788527153643982e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98407281 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173086524551376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178923328465408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44329782 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12905013 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54077501 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44329780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0;j² ò'ÍìO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 $ _timestamp‚1.788527153643982e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98407281 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173086524551376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178923328465408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44329782 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12905013 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54077501 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44329780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚2149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178923310462464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2434037226517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2443179946496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54103843 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893¬þûJÝ2Î- _wandbruntime‚2538  _runtime‚2538Ï/Q&ÏòM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54128700 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197641336594432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181448320984576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443391436288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181449248971264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2149 _runtime‚2538.175026067 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44349940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175599771631359 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179004929548288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211992576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179004947551232 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200199284529152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642î·ëý&Ð÷M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173165542145920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235932672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197641360534528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98478642 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39472569 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44349940 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885271546380353e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173165525106560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12911298 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200199284529152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54128700 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197641336594432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181448320984576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443391436288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181449248971264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2149 _runtime‚2538.175026067 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44349940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175599771631359 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179004929548288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211992576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179004947551232 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183004264753664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2498357401088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185503550128128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55322403 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40349804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179526393745845 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197348 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179527318760201 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204678065446400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202062420052480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615645393920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40349196 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202062396112384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204677827403264 _runtime‚2595.816023817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13196742 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0FL2{ áñO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204678065446400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202062420052480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615645393920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40349196 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202062396112384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204677827403264 _runtime‚2595.816023817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13196742 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183004264753664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2498357401088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185503550128128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55322403 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40349804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179526393745845 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197348 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179527318760201 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004°5É2â- _wandbruntime‚2596  _runtime‚2596NÜV²ü&ãöM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2491031755111 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45357942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204769472651264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204769222038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185586290687488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183085901842432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2490123779907 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2499478861312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2500388845056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45357942 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100731544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13203177 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45357940 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55347234 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2499478861312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179607453628231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27164521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185585362700800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55347234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55373602 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚177 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45357940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 _runtime‚2596.895261871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40367698 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2499478861312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179607453628231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27164521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185585362700800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55347234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55373602 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚177 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45357940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 _runtime‚2596.895261871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40367698 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45357942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204769472651264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204769222038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185586290687488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183085901842432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2490123779907 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2499478861312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2500388845056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2199±ÎÙ£2å- _wandbruntime‚2596  _runtime‚2596dž¬ÇVæÐ -loss/FRt1/detection_loss‚3.15264861230497 _step‚178 .loss/FRt4/detection_loss‚2.502743648928265 1loss/SRqqt1/detection_loss‚2.3886169502812047 1loss/SRqqt4/detection_loss‚1.3870073015967461 .loss/FBt1/detection_loss‚3.039512480298678 /loss/FBt3/detection_loss‚0.6590196179641209 /loss/FBt4/detection_loss‚3.1110460062821708 %loss/total_loss‚46.63372481111686 0loss/FBt4/assignment_loss‚0.4702462417638647 1loss/FBt3/assignment_loss‚0.12822482490368037 /loss/FBt2/detection_loss‚2.2169458735734224 % _timestamp‚1.7885272133611758e+09 /loss/FRt2/assignment_loss‚4.033605320700284 /loss/FRt4/assignment_loss‚3.248200293197188 2loss/SRqqt1/assignment_loss‚1.7910184730445184 0loss/FBt2/assignment_loss‚0.4228487161162775 0loss/FRt3/assignment_loss‚1.2026381318767865 .loss/FRt2/detection_loss‚2.732909054591738 1loss/SRqqt2/detection_loss‚2.5778520266215006 /loss/FRt1/assignment_loss‚3.525910770451581 2loss/SRqqt2/assignment_loss‚1.7469234228134156 1loss/SRqqt4/assignment_loss‚1.342415540198451 /loss/FRt3/detection_loss‚2.0147857293486595 epoch‚1 _runtime‚2596.896534356 2loss/SRqqt3/assignment_loss‚1.3443729661761181 0loss/FBt1/assignment_loss‚0.3536437195968271 0loss/SRqqt3/detection_loss‚1.240589088486398 trainer/global_step‚2199IE¯z[çÕ 1loss/SRqqt4/detection_loss‚1.3870073015967461 .loss/FBt1/detection_loss‚3.039512480298678 /loss/FBt3/detection_loss‚0.6590196179641209 /loss/FBt4/detection_loss‚3.1110460062821708 %loss/total_loss‚46.63372481111686 0loss/FBt4/assignment_loss‚0.4702462417638647 1loss/FBt3/assignment_loss‚0.12822482490368037 /loss/FBt2/detection_loss‚2.2169458735734224 % _timestamp‚1.7885272133611758e+09 /loss/FRt2/assignment_loss‚4.033605320700284 /loss/FRt4/assignment_loss‚3.248200293197188 2loss/SRqqt1/assignment_loss‚1.7910184730445184 0loss/FBt2/assignment_loss‚0.4228487161162775 0loss/FRt3/assignment_loss‚1.2026381318767865 .loss/FRt2/detection_loss‚2.732909054591738 1loss/SRqqt2/detection_loss‚2.5778520266215006 /loss/FRt1/assignment_loss‚3.525910770451581 2loss/SRqqt2/assignment_loss‚1.7469234228134156 1loss/SRqqt4/assignment_loss‚1.342415540198451 /loss/FRt3/detection_loss‚2.0147857293486595 epoch‚1 _runtime‚2596.896534356 2loss/SRqqt3/assignment_loss‚1.3443729661761181 0loss/FBt1/assignment_loss‚0.3536437195968271 0loss/SRqqt3/detection_loss‚1.240589088486398 trainer/global_step‚2199 -loss/FRt1/detection_loss‚3.15264861230497 _step‚178 .loss/FRt4/detection_loss‚2.502743648928265 1loss/SRqqt1/detection_loss‚2.3886169502812047²§Ú…”è‚(:‰ ú„ëÔÌÝá gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r‹Ñ >é‚(:³ ú„ëÔøÜâ disk.rbd0.in‚0) disk./.usagePercent‚75.90140532930629 disk.nvme0n1.in‚0 disk.sdb.out‚776.5966796875* proc.memory.percent‚18.190574645996094 cpu‚25.137680548651502 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 765.77734375 disk.md0p1.out‚ 765.77734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 70.9609375 disk.sda2.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.out‚ 27598.53125 disk.rbd0.out‚ 0.91259765625 proc.memory.rssMB‚ 2980.34375 network.sent‚ 4.378353e+06 network.recv‚929163 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda.out‚776.5966796875! disk.nvme1n1.out‚ 13220.609375$ memory_percent‚99.95729923248291& proc.memory.availableMB‚ 6.99609375 disk.dm-0.in‚0 disk.dm-0.out‚ 40819.140625% disk./.usageGB‚2714.6019020080566 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.out‚776.5966796875 disk.md0.in‚ 199.5859375 disk.sdb2.out‚776.5966796875 disk.nvme1n1.in‚0ÅÚî3”ê‚(:‰ ‰…ëÔû¥Ïà gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G›¶¿ë‚(:µ ‰…ëÔ”‡©á$ disk./.usageGB‚2714.638946533203 disk.rbd0.out‚ 0.91259765625 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sda.out‚777.88818359375 cpu‚25.13682273268851 network.recv‚936609! disk.sdb2.out‚777.88818359375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.0625! disk.sda2.out‚777.88818359375 disk.md0.in‚ 199.5859375 disk.dm-0.out‚41002.95703125 disk.sda.in‚128.625& proc.memory.availableMB‚ 6.99609375 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0$ memory_percent‚99.95729923248291 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚27708.17578125 disk.dm-0.i¿ Ò¡n‚0 proc.memory.rssMB‚ 2980.34375 proc.cpu.threads‚17 disk.md0.out‚ 767.015625) disk./.usagePercent‚75.90244110973372 disk.sdb.out‚777.88818359375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 767.015625* proc.memory.percent‚18.190574645996094 network.sent‚ 4.486152e+06 disk.nvme1n1.out‚ 13294.78125 disk.sda1.out‚0 disk.nvme0n1.in‚0Âîkv“ì‚(:ˆ ˜…ëÔГ ß gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w™˜©9í‚(:® ˜…ëÔû§žá proc.cpu.threads‚17 disk.rbd2.out‚1.0625 network.recv‚940905 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 779.05859375 disk.sda.in‚128.625 disk.md0p1.out‚ 768.12109375& proc.memory.availableMB‚ 6.71484375) disk./.usagePercent‚75.90593445548177 disk.md0.in‚ 199.5859375 disk.dm-0.out‚41343.26953125# disk.nvme1n1.out‚13372.67578125 disk.sdb1.out‚0 cpu‚25.136053881248657 disk.sda.out‚ 779.05859375* proc.memory.percent‚18.190574645996094 network.sent‚ 4.494454e+06$ disk./.usageGB‚2714.763885498047 disk.dm-0.in‚0 disk.nvme0n1.out‚ 27970.59375 proc.memory.rssMB‚ 2980.34375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚ 779.05859375 disk.md0p1.in‚ 199.5859375 disk.sdb.out‚ 779.05859375 disk.sda1.in‚0$ memory_percent‚99.95901584625244 disk.md0.out‚ 768.12109375 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.91259765625 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0<½Ý2î- _wandbruntime‚2652  _runtime‚2652?ÁìhÇïÁ & lr-AdamW/pg1‚0.0007154277160765696 & lr-AdamW/pg2‚0.0007154277160765696 trainer/global_step‚2249 % _timestamp‚1.7885272686808789e+09 _runtime‚2652.216394066 _step‚179aÖ¾†ÌðÆ & lr-AdamW/pg1‚0.0007154277160765696 & lr-AdamW/pg2‚0.0007154277160765696 trainer/global_step‚2249 % _timestamp‚1.7885272686808789e+09 _runtime‚2652.216394066 _step‚179³¤Íï82ñ- _wandbruntime‚2652  _runtime‚2652—œoñ'òëO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41244223 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2554444826112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27755706 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2544883761737 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56567305 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209248056630272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46345782 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885272686816673e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674122572288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _runtime‚ 2652.21728743 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545791736733 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56593647 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2555354796544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187085219044864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180988266966416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46345780 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56567305 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 trainer/global_step‚2249¥Tùãö'óðO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885272686816673e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674122572288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _runtime‚ 2652.21728743 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545791736733 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56593647 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2555354796544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187085219044864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180988266966416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46345780 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56567305 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 trainer/global_step‚2249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41244223 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2554444826112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27755706 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2544883761737 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56567305 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209248056630272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46345782 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784´è_$õ2ô- _wandbruntime‚2653  _runtime‚2653Ι¯V&õöM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27767771 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183613285595975 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2653.196525843 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚206663907039232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555566286336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102958076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46365940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546001035015 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187166838130688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189723332403712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56592136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206663930979328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189723332403712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181067301600320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚2249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27767769 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56618504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187166856133632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56592136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2556476270080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181067284560960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546909010219 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2653.196525843 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚206663907039232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555566286336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102958076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46365940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546001035015 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187166838130688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189723332403712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56592136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206663930979328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189723332403712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181067301600320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚2249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27767769 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56618504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187166856133632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56592136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2556476270080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181067284560960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546909010219 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187539907710461 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47353782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2733053980160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2610532251136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13782064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47353782 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57812207 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2299 _step‚184 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13781443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788527324507497e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13781443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMoéS6¾4nitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788527324507497e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187539907710461 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47353782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2733053980160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2610532251136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13782064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47353782 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57812207 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2299 _step‚184 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193860374119936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885273255743487e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚2299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13787214 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28371021 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191247810424832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213909211374080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185018181327520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193860374119936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2612563695104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚2709.110748609 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191247792421888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28371019 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42158855 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213909211374080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185018181327520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193860374119936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2612563695104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚2709.110748609 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191247792421888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28371019 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42158855 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193860374119936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885273255743487e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚2299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13787214 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¹—2»_2‹- _wandbruntime‚2709  _runtime‚2709äÈYŒÓ 2loss/SRqqt3/assignment_loss‚1.0876648060275593 1loss/FBt4/assignment_loss‚0.19196286774240434 0loss/SRqqt2/detection_loss‚2.666587096673471 0loss/FBt2/assignment_loss‚0.6527115200951812 0loss/SRqqt1/detection_loss‚2.939008911450704 /loss/FBt2/detection_loss‚2.2487467879598793 2loss/SRqqt4/assignment_loss‚1.3884210221525004 /loss/FRt3/detection_loss‚1.5187062646448612 /loss/FBt3/detection_loss‚0.7860295420694279 _step‚186 /loss/FRt3/assignment_loss‚1.341934186220169 2loss/SRqqt2/assignment_loss‚1.6946076427896817 1loss/FBt3/assignment_loss‚0.17442535713116297 -loss/FRt2/detection_loss‚2.59393118818601 0loss/SRqqt4/detection_loss‚1.348663444227692 &loss/total_loss‚45.623387997691346 0loss/FRt4/assignment_loss‚3.1897153119246164 1loss/SRqqt3/detection_loss‚1.1454546046645746 epoch‚1 _runtime‚2709.112417897 .loss/FRt1/detection_loss‚2.778933572769165 /loss/FBt1/detection_loss‚2.6085658113161725 /loss/FBt4/detection_loss‚2.4571553667386374 trainer/global_step‚2299 % _timestamp‚1.7885273255764298e+09 .loss/FRt1/assignment_loss‚4.54247795343399 /loss/FRt2/assignment_loss‚4.277503275209003 1loss/FBt1/assignment_loss‚0.17373680056104301 2loss/SRqqt1/assignment_loss‚1.8851170353591442 /loss/FRt4/detection_loss‚1.9313276283442975ë}g^Ø 2loss/SRqqt3/assignment_loss‚1.0876648060275593 1loss/FBt4/assignment_loss‚0.19196286774240434 0loss/SRqqt2/detection_loss‚2.666587096673471 0loss/FBt2/assignment_loss‚0.6527115200951812 0loss/SRqqt1/detection_loss‚2.939008911450704 /loss/FBt2/detection_loss‚2.2487467879598793 2loss/SRqqt4/assignment_loss‚1.3884210221525004 /loss/FRt3/detection_loss‚1.5187062646448612 /loss/FBt3/detection_loss‚0.7860295420694279 _step‚186 /loss/FRt3/assignment_loss‚1.341934186220169 2loss/SRqqt2/assignment_loss‚1.6946076427896817 1loss/FBt3/assignment_loss‚0.17442535713116297 -loss/FRt2/detection_loss‚2.59393118818601 0loss/SRqqt4/detection_loss‚1.348663444227692 &loss/total_loss‚45.623387997691346 0loss/FRt4/assignment_loss‚3.1897153119246164 1loss/SRqqt3/detection_loss‚1.1454546046645746 epoch‚1 _runtime‚2709.112417897 .loss/FRt1/detection_loss‚2.778933572769165 /loss/FBt1/detection_loss‚2.6085658113161725 /loss/FBt4/detection_loss‚2.4571553667386374 trainer/global_step‚2299 % _timestamp‚1.7885273255764298e+09 .loss/FRt1/assignment_loss‚4.54247795343399 /loss/FRt2/assignment_loss‚4.277503275209003 1loss/FBt1/assignment_loss‚0.17373680056104301 2loss/SRqqt1/assignment_loss‚1.8851170353591442 /loss/FRt4/detection_loss‚1.9313276283442975º>#±’Ž‚(:‡ ã…ëÔÍž§à gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Fcý?‚(:´ ã…ëÔ¹ž€á disk.nvme0n1.in‚0 disk.dm-0.out‚42547.87890625 proc.memory.rssMB‚ 2980.34375* proc.memory.percent‚18.190574645996094% disk./.usageGB‚2714.6116065979004 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚786.83935546875! disk.sdb2.out‚786.83935546875 disk.md0p1.in‚ 199.5859375 cpu‚25.133063516266102 disk.dm-0.in‚0 disk.nvme1n1.in‚0 network.sent‚ 4.621734e+06 disk.rbd2.out‚1.0625" disk.nvme0n1.out‚ 28704.6796875! disk.sda2.out‚786.83935546875 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0% proc.memory.availableMB‚ 7.1015625 disk.sda2.in‚128.625 disk.md0p1.out‚ 775.609375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚963709 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 775.609375 disk.sda.in‚128.625 disk.sdb1.out‚0 disk.sda.out‚786.83935546875# disk.nvme1n1.out‚13843.19921875 disk.rbd0.out‚ 0.91259765625$ memory_percent‚99.95665550231934% disk./.usagePercent‚ 75.9016766737 disk.sda1.out‚0«t¤î”‚(:‰ ò…ëÔÙ’æà gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚149.165' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.29125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|•ôA‘‚(:³ ò…ëÔ‘ýá disk.rbd2.in‚0 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚13887.57421875$ memory_percent‚99.95660781860352) disk./.usagePercent‚75.90042106591586 disk.nvme0n1.out‚ 28966.9375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 network.recv‚969908 disk.md0p1.in‚ 199.5859375 disk.sda2.out‚788.0693359375 disk.md0.out‚ 776.79296875 disk.sda.in‚128.625* proc.memory.percent‚18.190574645996094 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sdb2.out‚788.0693359375 cpu‚25.132501700878468 disk.md0p1.out‚ 776.79296875 disk.dm-0.out‚42854.51171875 proc.memory.rssMB‚ 2980.34375 network.sent‚ 4.72897e+06% disk./.usageGB‚2714.5666999816895 d+¦Ê1<isk.rbd0.in‚0 disk.rbd0.out‚ 0.91259765625 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17 disk.sdb.out‚788.0693359375 disk.sda.out‚788.0693359375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.nvme1n1.in‚0$ proc.memory.availableMB‚7.109375g]IÒ“’‚(:ˆ †ëԽܓà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C’ìo:“‚(:¯ †ëÔÄëºà cpu‚25.131992954022074 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 42926.7734375% proc.memory.availableMB‚ 6.8359375 disk.rbd2.out‚1.0625 disk.nvme0n1.out‚ 28967.09375 disk.dm-0.in‚0 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375$ memory_percent‚99.95827674865723 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375 disk.sda.out‚789.26611328125 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2980.34375* proc.memory.percent‚18.190574645996094) disk./.usagePercent‚75.90244260298117$ disk./.usageGB‚2714.638999938965 disk.sdb.out‚789.26611328125! disk.sdb2.out‚789.26611328125 disk.md0p1.out‚777.9375 network.recv‚974716 disk.rbd0.out‚ 0.91259765625" disk.nvme1n1.out‚ 13959.6796875 disk.sda2.in‚128.625! disk.sda2.out‚789.26611328125 network.sent‚ 4.735062e+06 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚777.9375!ö²’”‚(:‡ †ëÔçŸã gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.79' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.9475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Ü•"D•‚(:¹ †ëÔ¥ìÐæ disk.dm-0.out‚43113.06640625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚791.17138671875" disk.nvme1n1.out‚ 13988.5546875 disk.md0p1.in‚ 199.5859375 cpu‚25.131526356391518! disk.sda2.out‚791.17138671875$ disk./.usageGB‚2714.764301300049 disk.rbd2.out‚1.0625 disk.sda.out‚791.17138671875# proc.memory.availableMB‚6.90625# disk.nvme0n1.out‚29124.51171875 disk.sda1.in‚0 network.sent‚ 4.743586e+06 disk.md0p1.out‚ 779.78515625 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚99.95784759521484 proc.cpu.threads‚17) disk./.usagePercent‚75.90594608147977 disk.rbd0.out‚ 0.97509765625 disk.sda1.out‚0 disk.sda.in‚128.625 proc.memory.rssMB‚ 2980.34375 disk.sdb1.out‚0 network.recv‚979348 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sdb.out‚791.17138671875* proc.memory.percent‚18.190574645996094 disk.rbd0.in‚0 disk.md0.out‚ 779.78515625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375ñï®2–- _wandbruntime‚2763  _runtime‚2763ýU¸Æ—À $ _timestamp‚1.788527380282763e+09 _runtime‚2763.818282817 _step‚187 & lr-AdamW/pg1‚0.0007121577519318452 & lr-AdamW/pg2‚0.0007121577519318452 trainer/global_step‚2349ä°È4˘Š_runtime‚2763.818282817 _step‚187 & lr-AdamW/pg1‚0.0007121577519318452 & lr-AdamW/pg2‚0.0007121577519318452 trainer/global_step‚2349 $ _timestamp‚1.788527380282763e+09»í]b2™- _wandbruntime‚2763  _runtime‚2763®öÚÌñ'šëO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195247145630208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59057109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14072341 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218388211002880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2656638271953 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657546246949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48361782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28962204 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 trainer/global_step‚2349 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191547589707125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59083451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218387960376832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247¿¬ö'›ðO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195247145630208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59057109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14072341 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218388211002880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2656638271953 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657546246949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48361782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28962204 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 trainer/global_step‚2349 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191547589707125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59083451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218387960376832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48381940 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191627724575155 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59108308 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2667741136384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191626799560591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14078199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segme–•Ý³Dnt.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48381940 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191627724575155 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59108308 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2667741136384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191626799560591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14078199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60302011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222958096761344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49369782 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2850778056704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242237440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2723617071616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚199328081918464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109671791 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43929919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2712515527061 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220107537001984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60328353 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202051716993024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49369780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14365082 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49369782 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2722707101184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14364465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2850559759360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 _runtime‚2819.431198741 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60302011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222958338998784¤|„ ­ñO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49369780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14365082 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49369782 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2722707101184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14364465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2850559759360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 _runtime‚2819.431198741 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60302011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222958338998784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199328081918464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60302011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222958096761344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49369782 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2850778056704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242237440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2723617071616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚199328081918464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109671791 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43929919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2712515527061 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220107537001984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60328353 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202051716993024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344À~2³“®‚(:ˆ ̆ëÔ؉Èá gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãlša;¯‚(:° ̆ëÔƒø¿â proc.cpu.threads‚17 disk.dm-0.out‚44234.48828125 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14428.35546875 disk.sdb1.in‚0# memory_percent‚99.9589204788208 network.sent‚ 4.866316e+06 network.recv‚998355 disk.sda2.out‚ 797.072265625 disk.sdb.out‚ 797.072265625 disk.sda.in‚128.625 disk.sdb1.out‚0* proc.memory.percent‚18.190574645996094 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.sdb2.out‚ 797.072265625% disk./.usageGB‚2715.1397094726562 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.md0p1.out‚ 785.4609375 disk.rbd0.out‚ 1.06201171875( disk./.usagePercent‚75.9164426511064 disk.sda1.in‚0" disk.nvme0n1.out‚ 29806.1328125 disk.sda2.in‚128.625 disk.md0.out‚ 785.4609375 disk.sda.out‚ 797.072265625 disk.rbd0.in‚0 proc.memory.rssMB‚ 2980.34375 cpu‚25.12935346142193 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375& proc.memory.availableMB‚ 6.73046875 disk.rbd2.out‚1.0625é# F2°- _wandbruntime‚2820  _runtime‚2820TPü&±öM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199409719007232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202133529565696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚2399 _step‚193 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2851951627264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192919923742560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49389942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199409701004288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49389940 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2724738545152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885274369318192e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2713632800339 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220197786646528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199409719007232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49389940 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60326842 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49389942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223049738273792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 _runtime‚2820.468422032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109743152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192919940781920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223049483466752 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199409701004288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2724738545152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60326842 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29577521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615ÿ0ì¼'²ûM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2724738545152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885274369318192e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2713632800339 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220197786646528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199409719007232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49389940 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60326842 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49389942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223049738273792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 _runtime‚2820.468422032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109743152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192919940781920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223049483466752 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199409701004288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2724738545152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60326842 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29577521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199409719007232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202133529565696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚2399 _step‚193 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2851951627264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192919923742560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49389942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199409701004288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49389940 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2ÁT¹ 2³- _wandbruntime‚2820  _runtime‚2820ã%G([´Õ 1loss/SRqqt1/assignment_loss‚2.149462665617466 1loss/FBt2/assignment_loss‚0.07068825037793301 .loss/FRt2/detection_loss‚2.579660679842975 /loss/FBt2/detection_loss‚3.1593546526772633 _runtime‚2820.469910249 0loss/FRt2/assignment_loss‚3.9973908066749573 0loss/FBt3/assignment_loss‚0.1549086333208278 .loss/FBt4/detection_loss‚2.559758947445796 $loss/total_loss‚46.8986539921032 trainer/global_step‚2399 % _timestamp‚1.7885274369345214e+09 1loss/FBt1/assignment_loss‚0.13665171845905205 /loss/FRt3/detection_loss‚1.6714497156820054 1loss/SRqqt3/detection_loss‚0.8339343678068232 /loss/FRt4/detection_loss‚2.1509119605316833 _step‚194 2loss/SRqqt2/assignment_loss‚2.0041016560580047 /loss/FRt1/detection_loss‚3.3648751699007473 /loss/FBt1/detection_loss‚2.5132699638605116 /loss/FBt3/detection_loss‚0.7482513575149434 1loss/SRqqt2/detection_loss‚2.3173741476876395 epoch‚1 /loss/FRt1/assignment_loss‚4.222192021516653 .loss/FRt4/assignment_loss‚3.50675504581601 2loss/SRqqt3/assignment_loss‚1.0178370522135756 2loss/SRqqt4/assignment_loss‚1.4275136192274445 1loss/FBt4/assignment_loss‚0.42020489003670475 1loss/SRqqt4/detection_loss‚1.9175403877800585 0loss/FRt3/assignment_loss‚1.5266695749961723 1loss/SRqqt1/detection_loss‚2.4478967070579527OÀid`µÚ /loss/FBt2/detection_loss‚3.1593546526772633 _runtime‚2820.469910249 0loss/FRt2/assignment_loss‚3.9973908066749573 0loss/FBt3/assignment_loss‚0.1549086333208278 .loss/FBt4/detection_loss‚2.559758947445796 $loss/total_loss‚46.8986539921032 trainer/global_step‚2399 % _timestamp‚1.7885274369345214e+09 1loss/FBt1/assignment_loss‚0.13665171845905205 /loss/FRt3/detection_loss‚1.6714497156820054 1loss/SRqqt3/detection_loss‚0.8339343678068232 /loss/FRt4/detection_loss‚2.1509119605316833 _step‚194 2loss/SRqqt2/assignment_loss‚2.0041016560580047 /loss/FRt1/detection_loss‚3.3648751699007473 /loss/FBt1/detection_loss‚2.5132699638605116 /loss/FBt3/detection_loss‚0.7482513575149434 1loss/SRqqt2/detection_loss‚2.3173741476876395 epoch‚1 /loss/FRt1/assignment_loss‚4.222192021516653 .loss/FRt4/assignment_loss‚3.50675504581601 2loss/SRqqt3/assignment_loss‚1.0178370522135756 2loss/SRqqt4/assignment_loss‚1.4275136192274445 1loss/FBt4/assignment_loss‚0.42020489003670475 1loss/SRqqt4/detection_loss‚1.9175403877800585 0loss/FRt3/assignment_loss‚1.5266695749961723 1loss/SRqqt1/detection_loss‚2.4478967070579527 1loss/SRqqt1/assignment_loss‚2.149462665617466 1loss/FBt2/assignment_loss‚0.07068825037793301 .loss/FRt2/detection_loss‚2.579660679842975Â*.Õö’¶‚(:‡ Û†ëÔ€¶£à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëñ OB·‚(:¹ Û†ëÔ”Œá disk.md0p1.in‚ 199.5859375$ memory_percent‚99.95896816253662 network.sent‚ 4.961789e+06$ disk./.usageGB‚2714.612163543701 disk.rbd0.in‚0 disk.rbd0.out‚ 1.06201171875# disk.nvme0n1.out‚30068.20703125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚798.3759765625 disk.md0.out‚ 786.703125 disk.sdb.in‚ 70.9609375& proc.memory.availableMB‚ 6.72265625 proc.memory.rssMB‚ 2980.34375 disk.sda1.in‚0 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 14472.0546875 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.dm-0.out‚44540.26171875 disk.md0p1.out‚ 786.703125* proc.memory.percent‚18.190574645996094 proc.cpu.threads‚17( disk./.usagePercent‚75.9016922461377 disk.sda1.out‚0 ­s£õdisk.sda.out‚798.3759765625 disk.sdb1.in‚0 disk.sda.in‚128.625 cpu‚25.128875637432913 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚798.3759765625 network.recv‚ 1.004097e+06 disk.md0.in‚ 199.5859375 disk.sdb.out‚798.3759765625þˆA¸‚(:¶ ê†ëÔÁÊŸà disk.sda.in‚128.625# disk.nvme1n1.out‚14472.30859375 network.recv‚ 1.010019e+06$ disk./.usageGB‚2714.637950897217 disk.rbd2.in‚0! disk.nvme0n1.out‚ 30196.828125 disk.dm-0.out‚44669.13671875 disk.sdb.in‚ 70.9609375 disk.sda.out‚800.15283203125 disk.sdb1.in‚0 disk.rbd0.in‚0 network.sent‚ 5.058555e+06 disk.sda1.out‚0 disk.dm-0.in‚0! disk.sdb2.out‚800.15283203125 disk.sda1.in‚0 disk.sdb.out‚800.15283203125* proc.memory.percent‚18.190574645996094 disk.md0p1.out‚ 788.42578125) disk./.usagePercent‚75.90241327133484 disk.sda2.in‚128.625 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2980.34375 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 788.42578125 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0$ memory_percent‚99.95861053466797# proc.memory.availableMB‚6.78125 disk.nvme0n1.in‚0! disk.sda2.out‚800.15283203125 disk.rbd0.out‚1.125 cpu‚25.142223360797875 proc.cpu.threads‚17IDÕ“¹‚(:ˆ ê†ëÔ‹„²à gpu.0.gpu‚64.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Ýžg’º‚(:‡ ù†ëÔð« à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AdÝzA»‚(:¶ ù†ëÔùÖžâ disk.dm-0.in‚0 disk.sda.out‚801.42822265625# disk.nvme1n1.out‚14472.44140625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.190574645996094) disk./.usagePercent‚75.90246201519801 disk.sdb1.in‚0! disk.sda2.out‚801.42822265625 disk.md0.in‚ 199.5859375 disk.sdb.out‚801.42822265625 disk.sda.in‚128.625$ proc.memory.availableMB‚6.796875 disk.rbd2.out‚1.0625" disk.nvme0n1.out‚ 30297.9296875 cpu‚25.163342222250755 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 789.6484375 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚801.42822265625$ memory_percent‚99.95851516723633 network.sent‚ 5.067011e+06 disk.sdb1.out‚0 disk.sda1.in‚0$ disk./.usageGB‚2714.639694213867 disk.rbd0.out‚1.125 disk.md0.out‚ 789.6484375 disk.nvme1n1.in‚0 network.recv‚ 1.014968e+06 disk.dm-0.out‚44770.37109375 proc.memory.rssMB‚ 2980.34375 disk.rbd2.in‚0 disk.sda2.in‚128.625ðí•“¼‚(:ˆ ˆ‡ëÔ¥£µà gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ¶i=½‚(:² ˆ‡ëÔéäæà network.sent‚ 5.070996e+06 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.90946641234245* proc.memory.percent‚18.190574645996094 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.rbd2.out‚1.0625$ disk./.usageGB‚2714.890205383301 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 cpu‚25.184046927844296 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.dm-0.out‚45056.38671875 disk.md0.in‚ 199.5859375 disk.sdb.out‚802.7568359375$ proc.memory.availableMB‚6.796875 disk.nvme1n1.in‚0 disk.rbd0.out‚1.125$ memory_percent‚99.95851516723633 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚802.7568359375 disk.md0.out‚ 790.9140625 network.recv‚ 1.019277e+06 disk.sdb2.out‚802.7568359375 disk.sda.out‚802.7568359375" disk.nvme1n1.out‚ 14472.6796875 disk.md0p1.out‚ 790.9140625# disk.nvme0n1.out‚30583.70703125 proc.memory.rssMB‚ 2980.34375_ú>’¾‚(:‡ —‡ëÔ—¬Áâ gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.364' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.591 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b„«~;¿‚(:° —‡ëÔ·é‹ã disk.md0p1.out‚ 792.43359375 disk.sda2.in‚128.625! disk.sda2.out‚804.33642578125 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 network.sent‚ 5.078494e+06 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb.out‚804.33642578125 disk.nvme0n1.in‚0 disk.sda.in‚128.625 disk.rbd0.out‚1.125) disk./.usagePercent‚75.90945201317061 disk.nvme1n1.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚804.33642578125% proc.memory.availableMB‚ 6.7421875* proc.memory.percent‚18.190574645996094! disk.nvme0n1.out‚ 30926.109375 disk.md0.out‚ 792.43359375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sda.out‚804.33642578125$ memory_percent‚99.95884895324707 proc.memory.rssMB‚ 2980.34375# disk./.usageGB‚2714.88969039917! disk.nvme1n1.out‚ 14472.828125 disk.rbd2.out‚1.0625 cpu‚25.20498527652938 network.recv‚ 1.023382e+06 disk.sdb1.out‚0 disk.dm-0.out‚ 45398.9375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0Wåû2À- _wandbruntime‚2900  _runtime‚2900zá‡Y*ÁÓT [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚14739129 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2768944711168 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206306591706624 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227476079696384 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038208 % _timestamp‚1.7885275169147758e+09 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚203537646995456 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚50511146 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206305890665472 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚50511144 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2757911416995 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2768944711168 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111743044 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767168 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227476276236288 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041152 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚45002138 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2768261672960 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 196539904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚203537628992512 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111722333 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚61211189 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚45002358 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚224578959462400 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2897144174080 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚203537646995456 _runtime‚2900.450723484õ;Ös^*ÂØT UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚50511146 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚195 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111722333 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206306591706624 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2897316773888 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416513 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2768944711168 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206306591706624 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227476079696384 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038208 % _timestamp‚1.7885275169147758e+09 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚203537646995456 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚50511146 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206305890665472 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚50511144 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2757911416995 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2768944711168 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111743044 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767168 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227476276236288 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041152 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚45002138 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2768261672960 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 196539904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚203537628992512 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111722333 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚61211189 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚45002358 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚224578959462400 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2897144174080 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚203537646995456 _runtime‚2900.450723484 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚14739129 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 _step‚196 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚50526620 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚50526622 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206361342464512 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚61240712 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚ 2901.68599871 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚50526620 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767168 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 trainer/global_step‚2417 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 172599808 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/resÁ8Å”%erved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067440 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚199621945331431 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737344 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2769434562048 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111767334 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 _step‚196 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚50526620 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚50526622 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206361342464512 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚61240712 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚ 2901.68599871 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚50526620 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767168 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 trainer/global_step‚2417 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 172599808 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067440 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚199621945331431 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737344 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2769434562048 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111767334 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 B&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909089148044586 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14893551170825958 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37310898303985596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9768574833869934 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5281426310539246 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46842721104621887 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.493112176656723 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4659706652164459 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45517241954803467 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49723029136657715 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0028159813955426216 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8179379105567932 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1698882132768631 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1509493887424469 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49431711435317993 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34068140387535095 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9647452235221863 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9747406244277954 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.41160717606544495 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21720559895038605 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.43619275093078613 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15240037441253662 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949097633361816 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957298636436462 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 epoch‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21984784305095673 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_5‚1 ,particle/sensitivity‚0.28299322724342346 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5040690898895264 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_7‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3475247323513031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3879464566707611 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34903088212013245 +jet/accuracy_3_of_5‚0.07335509359836578 particle/accuracy_7_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4830247163772583 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32870373129844666 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3541666865348816 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4265340566635132 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8924680948257446 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18795983493328094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37483665347099304 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4426684081554413 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35421448945999146 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15848006308078766 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3541666865348816 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2172854244709015 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991493284702301 particle/accuracy_1_of_3‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9743631482124329 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1744173914194107 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03310033679008484 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46658921241760254 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42131808400154114 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_4‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4857730269432068 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24814820289611816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24649590253829956 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917200803756714 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45802193880081177 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40268608927726746 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23010098934173584 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16373825073242188 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697678923606Иå ÃY87 *jet/accuracy_2_of_5‚0.4322964549064636 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27342715859413147 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01840117946267128 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47686824202537537 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849259853363037 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4585033655166626 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21051791310310364 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21551884710788727 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887264966964722 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9713541865348816 ,jet/accuracy_3_of_3‚0.021273229271173477 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33436498045921326 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4840923547744751 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.150857076048851 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9539746046066284 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10192512720823288 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4527777135372162 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2747747600078583 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257572054862976 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9529966711997986 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4672619104385376 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004774788860231638 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6898631453514099 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4352683424949646 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21021680533885956 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4338420629501343 particle/accuracy_1_of_1‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.378976047039032 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0031746032182127237 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13845045864582062 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 'particle/f_score‚0.3681018054485321 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1699037104845047 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.185964897274971 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4663296341896057 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1862725019454956 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5134190917015076 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19405128061771393 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.471609503030777 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *jet/accuracy_1_of_2‚0.7390851378440857 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45447200536727905 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.45602044463157654 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 particle/accuracy_2_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35134080052375793 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.024395694956183434 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.486415296792984 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3920350968837738 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08972906321287155 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07557268440723419 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10905348509550095 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022041277959942818 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 particle/accuracy_3_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4050510823726654 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_6‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836654663085938 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.36141878366470337 *jet/accuracy_2_of_6‚0.5320513248443604 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47639569640159607 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11099401861429214 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20187082886695862 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15181808173656464 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3264263868331909 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.2666666805744171 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01708337664604187 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2739124596118927 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15558552742004395 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012969834730029106 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1509493887424469 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23687078058719635 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9819575548171997 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06011905148625374 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20679013431072235 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4860857129096985 particle/accuracy_5_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _step‚197 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4854823350906372 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3163575530052185 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.961904764175415 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46436890959739685 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4712488651275635 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07062524557113647 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1625729203224182 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.356337308883667 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37581515312194824 +jet/accuracy_2_of_3‚0.23424026370048523 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5313366055488586 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16299064457416534 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.40625 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27146464586257935 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2301587164402008 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46658921241760254 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_6‚1 % _timestamp‚1.7885275338996305e+09 )loss/val_total_loss‚47.17119896211086 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02036990039050579 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2911709249019623 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.028325123712420464 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 trainer/global_step‚2417 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902779459953308 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36094990372657776 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4061771631240845 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 *jet/accuracy_1_of_3‚0.8556656241416931 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 'particle/accuracy‚0.848395049571991 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.493805468082428 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2077586054801941 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_4_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17265227437019348 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.51932293176651 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20962148904800415 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22887760400772095 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08748965710401535 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20143115520477295 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3486473560333252 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9710692167282104 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32222220301628113 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.345370352268219 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9696711897850037 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 +"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.75 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4640568196773529 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3415178656578064 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2505323588848114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873377084732056 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_4‚0.05907641723752022 particle/accuracy_4_of_5‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 jet/accuracy_2_of_7‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4947916567325592 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚2917.437549255 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9758620262145996 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01308243814855814 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12817543745040894 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5063378810882568 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49367496371269226 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15123742818832397 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020198196172714233 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009214706718921661 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2097560167312622 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4347425401210785 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16762128472328186 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44908347725868225 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3762403428554535 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22131510078907013 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4477699100971222 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02372134104371071ÐŽË“/&Ê„¦ ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0031746032182127237 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13845045864582062 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 'particle/f_score‚0.3681018054485321 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1699037104845047 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.185964897274971 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4663296341896057 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1862725019454956 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5134190917015076 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19405128061771393 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.471609503030777 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *jet/accuracy_1_of_2‚0.7390851378440857 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45447200536727905 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.45602044463157654 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 particle/accuracy_2_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35134080052375793 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.024395694956183434 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.486415296792984 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3920350968837738 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08972906321287155 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07557268440723419 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10905348509550095 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022041277959942818 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 particle/accuracy_3_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4050510823726654 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_6‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836654663085938 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.36141878366470337 *jet/accuracy_2_of_6‚0.5320513248443604 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47639569640159607 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11099401861429214 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20187082886695862 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15181808173656464 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3264263868331909 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.2666666805744171 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01708337664604187 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2739124596118927 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15558552742004395 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012969834730029106 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1509493887424469 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23687078058719635 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9819575548171997 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06011905148625374 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20679013431072235 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4860857129096985 particle/accuracy_5_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _step‚197 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4854823350906372 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3163575530052185 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.961904764175415 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46436890959739685 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4712488651275635 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07062524557113647 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1625729203224182 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.356337308883667 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37581515312194824 +jet/accuracy_2_of_3‚0.23424026370048523 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5313366055488586 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16299064457416534 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.40625 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27146464586257935 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2301587164402008 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46658921241760254 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_6‚1 % _timestamp‚1.7885275338996305e+09 )loss/val_total_loss‚47.17119896211086 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02036990039050579 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2911709249019623 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.028325123712420464 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 trainer/global_step‚2417 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902779459953308 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36094990372657776 A&Purity/3FRt4SRqqt*FBt/event_p0y÷Ülroportion‚0.00012400794366840273 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4061771631240845 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 *jet/accuracy_1_of_3‚0.8556656241416931 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 'particle/accuracy‚0.848395049571991 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.493805468082428 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2077586054801941 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_4_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17265227437019348 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.51932293176651 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20962148904800415 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22887760400772095 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08748965710401535 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20143115520477295 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3486473560333252 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9710692167282104 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32222220301628113 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.345370352268219 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9696711897850037 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 +"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.75 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4640568196773529 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3415178656578064 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2505323588848114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873377084732056 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_4‚0.05907641723752022 particle/accuracy_4_of_5‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 jet/accuracy_2_of_7‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4947916567325592 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚2917.437549255 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9758620262145996 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01308243814855814 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12817543745040894 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5063378810882568 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49367496371269226 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15123742818832397 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020198196172714233 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009214706718921661 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2097560167312622 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4347425401210785 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16762128472328186 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44908347725868225 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3762403428554535 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22131510078907013 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4477699100971222 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02372134104371071 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36944445967674255 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6294381618499756 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.052229736000299454 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.49891507625579834 *jet/accuracy_1_of_6‚0.9615384340286255 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42006638646125793 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9857359528541565 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0632125660777092 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910678267478943 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28477010130882263 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46748700737953186 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_4‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3766666650772095 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1798364520072937 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4341435134410858 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968041777610779 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4385964870452881 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3387944996356964 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34639376401901245 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909089148044586 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14893551170825958 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37310898303985596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9768574833869934 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5281426310539246 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46842721104621887 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.493112176656723 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4659706652164459 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45517241954803467 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49723029136657715 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0028159813955426216 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8179379105567932 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1698882132768631 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1509493887424469 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49431711435317993 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34068140387535095 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9647452235221863 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9747406244277954 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.41160717606544495 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21720559895038605 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.43619275093078613 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15240037441253662 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949097633361816 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957298636436462 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 epoch‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21984784305095673 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_5‚1 ,particle/sensitivity‚0.28299322724342346 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5040690898895264 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_7‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3475247323513031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3879464566707611 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34903088212013245 +jet/accuracy_3_of_5‚0.07335509359836578 particle/accuracy_7_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4830247163772583 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32870373129844666 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3541666865348816 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4265340566635132 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8924680948257446 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18795983493328094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37483665347099304 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4426684081554413 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35421448945999146 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15848006308078766 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3541666865348816 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2172854244709015 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991493284702301 particle/accuracy_1_of_3‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9743631482124329 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1744173914194107 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03310033679008484 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46658921241760254 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42131808400154114 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_4‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4857730269432068 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24814820289611816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24649590253829956 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917200803756714 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45802193880081177 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40268608927726746 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23010098934173584 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16373825073242188 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4069767892360687 *jet/accuracy_2_of_5‚0.4322964549064636 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27342715859413147 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01840117946267128 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47686824202537537 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849259853363037 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4585033655166626 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21051791310310364 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21551884710788727 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887264966964722 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9713541865348816 ,jet/accuracy_3_of_3‚0.021273229271173477 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33436498045921326 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4840923547744751 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.150857076048851 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9539746046066284 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10192512720823288 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4527777135372162 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2747747600078583 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257572054862976 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9529966711997986 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4672619104385376 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004774788860231638 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6898631453514099 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4352683424949646 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21021680533885956 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4338420629501343 particle/accuracy_1_of_1‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.378976047039032 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408Å|ªU’Ë‚(:‡ µ‡ëÔÅÜžá gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔèªpDÌ‚(:¹ µ‡ëÔݹ¹á disk.sda.out‚807.60302734375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0% disk./.usageGB‚2716.1431465148926 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 proc.memory.rssMB‚2987.375) disk./.usagePercent‚75.94449917077836 disk.md0.in‚ 199.5859375 network.sent‚ 5.201581e+06 disk.rbd2.in‚0" disk.nvme0n1.out‚ 31817.9296875 disk.md0.out‚ 795.5703125) proc.memory.availableMB‚ 1315.58203125 cpu‚25.23616150440436 network.recv‚ 1.033491e+06 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.nvme0n1.in‚0! disk.sda2.out‚807.60302734375 disk.dm-0.in‚0 disk.dm-0.out‚46291.33203125# disk.nvme1n1.out‚14473.40234375 disk.md0p1.out‚ 795.5703125$ memory_percent‚91.97032451629639* proc.memory.percent‚18.233489990234375 disk.rbd0.out‚1.125 proc.cpu.threads‚17 disk.sdb.out‚807.60302734375! disk.sdb2.out‚807.60302734375 disk.sda.in‚128.625 disk.rbd2.out‚1.0625{Tp]AÍ‚(:¶ ćëÔëîµà disk.md0p1.out‚ 796.953125 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚128.625 disk.md0.out‚ 796.953125 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0 network.sent‚ 5.314101e+06$ disk./.usageGB‚2715.612335205078 disk.md0.in‚ 199.5859375 disk.sdb.out‚809.04736328125$ memory_percent‚91.96672439575195! proc.memory.rssMB‚ 2986.390625 cpu‚25.2350690762638 disk.nvme0n1.in‚0 disk.rbd0.out‚1.125 disk.sda1.out‚0) disk./.usagePercent‚75.92965746439415 disk.dm-0.out‚47536.77734375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚809.04736328125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 14473.6171875' proc.memory.availableMB‚ 1316.171875 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 network.recv‚ 1.040515e+06# disk.nvme0n1.out‚33065.53515625! disk.sda2.out‚809.04736328125 disk.dm-0.in‚0 disk.rbd2.in‚0 disk.sda.out‚809.04736328125 disk.sdb1.in‚0 disk.rbd2.out‚1.0625* proc.memory.percent‚18.227481842041016£ŸÃT“΂(:ˆ ćëÔµ§¦à gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÓ2}“Ï‚(:ˆ Ó‡ëÔ£ýÜÞ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É8ÃBЂ(:· Ó‡ëÔò„òß! proc.memory.rssMB‚ 2986.390625 disk.sdb1.out‚0 disk.dm-0.in‚0# memory_percent‚91.9670820236206* proc.memory.percent‚18.227481842041016 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 798.609375 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 47706.53125! disk.sdb2.out‚810.77001953125) disk./.usagePercent‚75.92964989149638% disk./.usageGB‚2715.6120643615723! disk.sda2.out‚810.77001953125 disk.md0.out‚ 798.609375 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 cpu‚25.23397867405697 proc.cpu.threads‚17 network.recv‚ 1.044535e+06 disk.rbd0.in‚0 disk.sda2.in‚128.625 disk.sda.in‚128.625# disk.nvme1n1.out‚14473.76953125) proc.memory.availableMB‚ 1316.11328125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.0625# disk.nvme0n1.out‚33232.76171875 disk.sda1.in‚0 disk.sda.out‚810.77001953125 network.sent‚ 5.319487e+06 disk.rbd0.out‚1.125 disk.sdb.out‚810.77001953125ž‰V2Ñ- _wandbruntime‚2958  _runtime‚2958ó~þKÇÒÁ & lr-AdamW/pg1‚0.0007085991642201604 & lr-AdamW/pg2‚0.0007085991642201604 trainer/global_step‚2449 % _timestamp‚1.7885275747171981e+09 _runtime‚2958.252659405 _step‚198‰/Å™fÓÆ & lr-AdamW/pg2‚0.0007085991642201604 trainer/global_step‚2449 % _timestamp‚1.7885275ŒoÈŠf747171981e+09 _runtime‚2958.252659405 _step‚198 & lr-AdamW/pg1‚0.0007085991642201604ÆýP2Ô- _wandbruntime‚2958  _runtime‚2958ÆJÛ>ò'ÕìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209638073444864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206827473351168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206827491354112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228163702053376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113483240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45762099 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30771110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 trainer/global_step‚2449 _runtime‚2958.253514298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45761464 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62130470 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2940346500096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62130470 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51352770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206827473351168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2810582090752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885275747179258e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209637145470976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62104127 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113456895 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51352770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2810582090752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2799167175231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚199 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2940589962752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231104292016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202767797677039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202768722691399 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199968647541168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345ý‡3>÷'ÖñO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14990354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51352768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62104127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231104024613376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30771110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 trainer/global_step‚2449 _runtime‚2958.253514298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45761464 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62130470 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2940346500096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62130470 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51352770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206827473351168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2810582090752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885275747179258e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209637145470976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62104127 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113456895 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51352770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2810582090752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2799167175231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚199 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2940589962752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231104292016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202767797677039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202768722691399 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199968647541168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209638073444864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206827473351168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206827491354112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228163702053376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113483240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45762099 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30783177 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206909110439936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209720814004224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51372928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202847932544861 _runtime‚2959.231391803 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62128958 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2941740799488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209719886017024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2811703564288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209720814004224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14996580 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231195403265024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 $ _timestamp‚1.788527575694687e+09¼ž ¬!ÙúM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45779757 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30783177 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206909110439936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209720814004224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.fre=Ußhßed‚51372928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202847932544861 _runtime‚2959.231391803 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62128958 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2941740799488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209719886017024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2811703564288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209720814004224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14996580 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231195403265024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 $ _timestamp‚1.788527575694687e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2810793580032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30783175 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228253927757824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0È$˜Hv2Ú- _wandbruntime‚2959  _runtime‚29596Ëñ ZÛÔ 0loss/FBt1/assignment_loss‚0.2538989612106282 1loss/SRqqt4/detection_loss‚1.8731159969018056 /loss/FBt4/detection_loss‚1.9765833467245102 2loss/SRqqt1/assignment_loss‚2.4140248397986093 .loss/FRt1/detection_loss‚2.933785406748454 1loss/SRqqt3/detection_loss‚0.9314239314326676 _step‚201 0loss/FRt1/assignment_loss‚3.6950009047985075 /loss/FRt4/assignment_loss‚3.273171064805011 0loss/FBt4/assignment_loss‚0.2193886698805727 1loss/SRqqt2/detection_loss‚2.3924740105867386 %loss/total_loss‚45.25862139780375 % _timestamp‚1.7885275756968446e+09 /loss/FRt3/detection_loss‚1.7748516878972833 /loss/FBt3/detection_loss‚0.6619724466030388 epoch‚2 _runtime‚2959.232134696 2loss/SRqqt3/assignment_loss‚1.1288642981840717 2loss/SRqqt4/assignment_loss‚1.6378772996294384 0loss/FBt2/assignment_loss‚0.3714659831019971 /loss/FRt4/detection_loss‚2.2619335388346595 1loss/SRqqt1/detection_loss‚2.3074660438757677 .loss/FBt1/detection_loss‚2.628506342569987 0loss/FRt3/assignment_loss‚1.3554013967514038 .loss/FRt2/detection_loss‚2.546475752806052 /loss/FRt2/assignment_loss‚3.861351456397619 1loss/FBt3/assignment_loss‚0.21843462871832153 /loss/FBt2/detection_loss‚2.7688009805149503 trainer/global_step‚2449 1loss/SRqqt2/assignment_loss‚1.772352409031656DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63375372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63375372 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46651114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52360768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865759544832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52360770 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52360770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15276754 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206774554659347 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63349029 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998673335808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2855952405339 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866669515776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235673636671488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866669515776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46651729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885276286046479e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115709797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31374362 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63375372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63375372 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46651114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52360768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865759544832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52360770 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52360770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15276754 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206774554659347 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63349029 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998673335808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2855952405339 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866669515776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235673636671488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866669515776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46651729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885276286046479e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115709797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31374362 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213857855720448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115781159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866881005056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885276297957923e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63400229 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2856161703617 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2857069678825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210990064731136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232765189040128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203998527823552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52380928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203998544862912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235765038711296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867790989312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15283051 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63373860 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31386427 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206855614541737 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15282442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235765287226368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185Ň•¦'ëûM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31386427 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206855614541737 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15282442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235765287226368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31386425 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213857855720448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52380930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232765212980224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52380928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210990046728192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213857855720448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115781159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866881005056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885276297957923e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63400229 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2856161703617 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2857069678825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210990064731136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232765189040128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203998527823552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52380928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203998544862912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235765038711296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867790989312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15283051 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63373860ÌkhëÒ2ì- _wandbruntime‚3013  _runtime‚3013Ü…AZSíÍ 1loss/SRqqt3/assignment_loss‚0.895049431312241 .loss/FRt1/detection_loss‚3.230741526769555 1loss/SRqqt4/detection_loss‚1.5729734833423907 .loss/FBt3/detection_loss‚0.638661595353161 &loss/total_loss‚47.003139846890065 0loss/SRqqt1/assignment_loss‚2.25109230194773 1loss/FBt3/assignment_loss‚0.22943742409296625 0loss/SRqqt2/detection_loss‚2.613356369572717 .loss/FBt1/detection_loss‚3.187068782746792 _step‚205 1loss/SRqqt4/assignment_loss‚1.398571751696559 0loss/FBt1/assignment_loss‚0.2549942745513931 1loss/FBt4/assignment_loss‚0.20676148965139873 .loss/FRt3/detection_loss‚1.453920221057805 1loss/SRqqt3/detection_loss‚0.8876303786313847 _runtime‚3013.338100358 0loss/FRt4/assignment_loss‚2.5175867152625115 1loss/FBt2/assignment_loss‚0.15228677503598323 .loss/FRt2/detection_loss‚2.888731117601748 0loss/SRqqt1/detection_loss‚2.527870824294431 .loss/FBt2/detection_loss‚3.051393755844661 /loss/FRt2/assignment_loss‚4.716792671768753 .loss/FRt4/detection_loss‚2.074699968870344 epoch‚2 trainer/global_step‚2499 % _timestamp‚1.7885276298004494e+09 0loss/FRt3/assignment_loss‚1.5092572450637818 1loss/SRqqt2/assignment_loss‚1.346283861612146 /loss/FRt1/assignment_loss‚4.359261087749315 /loss/FBt4/detection_loss‚3.0387167930603027–J×oXîÒ 1loss/SRqqt4/detection_loss‚1.5729734833423907 .loss/FBt3/detection_loss‚0.638661595353161 &loss/total_loss‚47.003139846890065 0loss/SRqqt1/assignment_loss‚2.25109230194773 1loss/FBt3/assignment_loss‚0.22943742409296625 0loss/SRqqt2/detection_loss‚2.613356369572717 .loss/FBt1/detection_loss‚3.187068782746792 _step‚205 1loss/SRqqt4/assignment_loss‚1.398571751696559 0loss/FBt1/assignment_loss‚0.2549942745513931 1loss/FBt4/assignment_loss‚0.20676148965139873 .loss/FRt3/detection_loss‚1.453920221057805 1loss/SRqqt3/detection_loss‚0.8876303786313847 _runtime‚3013.338100358 0loss/FRt4/assignment_loss‚2.5175867152625115 1loss/FBt2/assignment_loss‚0.15228677503598323 .loss/FRt2/detection_loss‚2.888731117601748 0loss/SRqqt1/detection_loss‚2.527870824294431 .loss/FBt2/detection_loss‚3.051393755844661 /loss/FRt2/assignment_loss‚4.716792671768753 .loss/FRt4/detection_loss‚2.074699968870344 epoch‚2 trainer/global_step‚2499 % _timestamp‚1.7885276298004494e+09 0loss/FRt3/assignment_loss‚1.5092572450637818 1loss/SRqqt2/assignment_loss‚1.346283861612146 /loss/FRt1/assignment_loss‚4.359261087749315 /loss/FBt4/detection_loss‚3.0387167930603027 1loss/SRqqt3/assignment_loss‚0.895049431312241 .loss/FRt1/detection_loss‚3.230741526769555ÍEdk ’ï‚(:‡ ˆëÔå”à gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.724' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.681 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ÚÚ@ð‚(:µ ˆëÔìü¥á disk.sda1.out‚0( proc.memory.availableMB‚ 1316.0390625 cpu‚25.229992744221445 disk.sdb2.out‚ 816.509765625 disk.rbd0.out‚1.125* proc.memory.percent‚18.227481842041016 disk.sda.out‚ 816.509765625# disk.nvme1n1.out‚14474.68359375 disk.sda2.in‚128.625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 816.509765625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.nvme0n1.in‚0! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17 disk.md0p1.out‚ 804.08984375# disk.nvme0n1.out‚34116.83984375 disk.md0.in‚ 199.5859375 disk.md0.out‚ 804.08984375# memory_percent‚91.9675350189209 disk.md0p1.in‚ 199.5859375 network.recv‚ 1.064266e+06) disk./.usagePercent‚75.94443005475354% disk./.usageGB‚2716.1406745910645 disk.rbd0.in‚0 disk.rbd2.out‚1.0625 disk.sda2.out‚ 816.509765625 disk.dm-0.out‚ 48591.5234375 network.sent‚ 5.455037e+06 disk.sdb1.in‚0 disk.sda1.in‚0ÕŽÓO”ñ‚(:‰ žˆëÔ‰óŸà gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å#-4Bò‚(:· žˆëÔ’Çâà disk.sdb.out‚817.57861328125! disk.sdb2.out‚817.57861328125" disk.nvme1n1.out‚ 14475.0703125 network.sent‚ 5.55167e+06 disk.md0p1.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 network.recv‚ 1.070821e+06 disk.md0.out‚ 805.09765625 disk.sda.out‚817.57861328125 cpu‚25.22895421940736 disk.rbd2.out‚1.0625 disk.dm-0.out‚ 49009.1796875 disk.nvme1n1.in‚0# memory_percent‚91.9682264328003 disk.nvme0n1.in‚0) proc.memory.availableMB‚ 1315.92578125 proc.cpu.threads‚17 disk.md0p1.out‚ 805.09765625 disk.rbd0.out‚1.125! disk.nvme0n1.out‚ 34534.109375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625( disk./.usagePercent‚75.9444010430888 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda2.out‚817.57861328125! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016$ disk./.usageGB‚2716.139636993408 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0Ußÿ”ó‚(:‰ ­ˆëÔ„¥îà gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mcþBô‚(:· ­ˆëÔ°³‹á disk.nvme1n1.in‚0 disk.rbd0.out‚1.125 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625! disk.sda2.out‚818.93603515625! proc.memory.rssMB‚ 2986.390625 disk.rbd2.out‚1.0625$ memory_percent‚91.96527004241943 cpu‚25.22791113971625 network.recv‚ 1.075366e+06 disk.sdb.out‚818.93603515625 disk.md0.in‚ 199.5859375) proc.memory.availableMB‚ 1316.41015625 network.sent‚ 5.55565e+06% disk./.usageGB‚2715.6123390197754 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚818.93603515625 disk.sda1.in‚0) disk./.usagePercent‚75.92965757105469 disk.dm-0.out‚ 49009.484375 disk.sdb1.out‚0 disk.nvme0n1.out‚ 34534.21875 disk.md0.out‚ 806.40234375 disk.sda.out‚818.93603515625! disk.nvme1n1.out‚ 14475.265625 disk.rbd0.in‚0 disk.sdb1.in‚0* proc.memory.percent‚18.227481842041016 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 806.40234375 disk.sda1.out‚0ÎhªÔ’õ‚(:‡ ¼ˆëÔÿË™à gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iµ5;ö‚(:° ¼ˆëÔÕÃÐà% disk./.usageGB‚2715.6120681762695 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0" proc.memory.availableMB‚1316.5 disk.dm-0.out‚ 49263.2734375 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 disk.sda.out‚820.81005859375 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.nvme0n1.in‚0! disk.sda2.out‚820.81005859375 proc.cpu.threads‚17 network.recv‚ 1.079593e+06 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14475.75390625 disk.rbd0.out‚1.125 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 808.21875( disk./.usagePercent‚75.9296499981569! disk.sdb2.out‚820.81005859375 disk.sda.in‚128.625 disk.sdb1.in‚0# disk.nvme0n1.out‚34787.51953125! proc.memory.rssMB‚ 2986.390625 cpu‚25.226810425084068 network.sent‚ 5.564012e+06 disk.sdb.out‚820.81005859375 disk.md0p1.out‚ 808.21875 disk.sdb1.out‚0# memory_percent‚91.9647216796875 disk.sdb2.in‚ 70.9609375Lí72÷- _wandbruntime‚3070  _runtime‚3070|¯i>ÇøÁ _step‚206 & lr-AdamW/pg1‚0.0007047549192043478 & lr-AdamW/pg2‚0.0007047549192043478 trainer/global_step‚2549 % _timestamp‚1.7885276871692905e+09 _runtime‚3070.704945943nñg–ùÆ trainer/global_step‚2549 % _timestamp‚1.7885276871692905e+09 _runtime‚3070.704945943 _step‚206 & lr-AdamW/pg1‚0.0007047549X"¹6192043478 & lr-AdamW/pg2‚0.0007047549192043478ÎÎä¡I2ú- _wandbruntime‚3070  _runtime‚3070øÛ„ò'ûìO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214989381933568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 _step‚207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47542381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214989399936512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217912156877312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2921846969856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚214989381933568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2911829660447 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64620274 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53368768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214989399936512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237186248558080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207870406995568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3057034265600 % _timestamp‚1.7885276871705837e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15564155 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31977612 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2549 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47541765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53368768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240243282823680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31977610 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 _runtime‚3070.706145448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3057254659584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0-*…÷'üñO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2549 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47541765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53368768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240243282823680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31977610 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 _runtime‚3070.706145448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3057254659584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214989381933568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 _step‚207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47542381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214989399936512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217912156877312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2921846969856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚214989381933568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2911829660447 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64620274 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53368768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214989399936512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237186248558080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207870406995568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3057034265600 % _timestamp‚1.7885276871705837e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15564155 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31977612 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640Ïå.«A2ý- _wandbruntime‚3071  _runtime‚30710n£Dû&þõM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64618762 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215071019022336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53388928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53388928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 _runtime‚3071.767427543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217994897436672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64645131 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2923878414336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118007690 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240334922243584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207949424590112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2922968430080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64645131 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207949407550752 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118034061 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240334661145600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210862371524045 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3058186883072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2912946933933 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2923878414336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237276474262528 $ _timestamp‚1.788527688230194e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ŸÞæQÿúM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64645131 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207949407550752 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118034061 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ë\q¯240334661145600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210862371524045 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3058186883072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2912946933933 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2923878414336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237276474262528 $ _timestamp‚1.788527688230194e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64618762 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215071019022336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53388928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53388928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 _runtime‚3071.767427543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217994897436672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64645131 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2923878414336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118007690 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240334922243584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207949424590112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2922968430080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760Ðòöè2€- _wandbruntime‚3071  _runtime‚30717¤÷YÓ 0loss/FRt4/assignment_loss‚3.6179303841458426 0loss/SRqqt3/assignment_loss‚1.29708324710804 1loss/SRqqt1/detection_loss‚2.7612573325634004 1loss/SRqqt2/detection_loss‚2.4658198139884253 0loss/FRt3/assignment_loss‚1.1256300806999207 1loss/SRqqt3/detection_loss‚0.9257058324637236 _runtime‚3071.769368932 0loss/FBt3/assignment_loss‚0.2537105528722214 1loss/SRqqt4/detection_loss‚1.8080065429210663 /loss/FBt3/detection_loss‚0.5985126090792021 epoch‚2 _step‚209 0loss/FRt2/assignment_loss‚4.1318685458256645 trainer/global_step‚2549 .loss/FRt1/assignment_loss‚3.76752508508748 1loss/SRqqt1/assignment_loss‚2.501992352679372 2loss/SRqqt4/assignment_loss‚1.5211403964459895 0loss/FBt1/assignment_loss‚0.2338352568403934 .loss/FRt1/detection_loss‚2.928349398333451 .loss/FRt2/detection_loss‚2.885846172387787 .loss/FRt4/detection_loss‚2.623660604159037 &loss/total_loss‚47.082604529829055 % _timestamp‚1.7885276882339664e+09 1loss/FBt4/assignment_loss‚0.22729939915091865 /loss/FBt1/detection_loss‚3.0798517167568207 1loss/FBt2/assignment_loss‚0.19534473812648553 .loss/FRt3/detection_loss‚1.538967671173234 /loss/FBt4/detection_loss‚2.4649447043736776 2loss/SRqqt2/assignment_loss‚1.6907767037099057 /loss/FBt2/detection_loss‚2.4375453889369965Z×è^‚Ø 2loss/SRqqt2/assignment_loss‚1.6907767037099057 /loss/FBt2/detection_loss‚2.4375453889369965 0loss/FRt4/assignment_loss‚3.6179303841458426 0loss/SRqqt3/assignment_loss‚1.29708324710804 1loss/SRqqt1/detection_loss‚2.7612573325634004 1loss/SRqqt2/detection_loss‚2.4658198139884253 0loss/FRt3/assignment_loss‚1.1256300806999207 1loss/SRqqt3/detection_loss‚0.9257058324637236 _runtime‚3071.769368932 0loss/FBt3/assignment_loss‚0.2537105528722214 1loss/SRqqt4/detection_loss‚1.8080065429210663 /loss/FBt3/detection_loss‚0.5985126090792021 epoch‚2 _step‚209 0loss/FRt2/assignment_loss‚4.1318685458256645 trainer/global_step‚2549 .loss/FRt1/assignment_loss‚3.76752508508748 1loss/SRqqt1/assignment_loss‚2.501992352679372 2loss/SRqqt4/assignment_loss‚1.5211403964459895 0loss/FBt1/assignment_loss‚0.2338352568403934 .loss/FRt1/detection_loss‚2.928349398333451 .loss/FRt2/detection_loss‚2.885846172387787 .loss/FRt4/detection_loss‚2.623660604159037 &loss/total_loss‚47.082604529829055 % _timestamp‚1.7885276882339664e+09 1loss/FBt4/assignment_loss‚0.22729939915091865 /loss/FBt1/detection_loss‚3.0798517167568207 1loss/FBt2/assignment_loss‚0.19534473812648553 .loss/FRt3/detection_loss‚1.538967671173234 /loss/FBt4/detection_loss‚2.4649447043736776Ñ$BD)”ƒ‚(:‰ ˈëÔ€®áâ gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šû%>;„‚(:° ˈëÔ¯å›å disk.dm-0.out‚49638.46484375 disk.sdb.out‚ 822.04296875$ memory_percent‚91.96572303771973( proc.memory.availableMB‚ 1316.3359375 proc.cpu.threads‚17 network.sent‚ 5.567198e+06 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚1.0625 disk.md0.out‚ 809.3984375 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sda.out‚ 822.04296875 disk.md0.in‚ 199.5859375# disk.nvme1n1.out‚14475.95703125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚ 822.04296875 disk.sdb2.out‚ 822.04296875 disk.md0p1.out‚ 809.3984375! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016) disk./.usagePercent‚75.93036355711679 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 cpu‚25.225876261643904 network.recv‚ 1.083504e+06" disk.nvme0n1.out‚ 35162.5078125 disk.rbd0.out‚1.125 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0% disk./.usageGB‚2715.6375885009766 disk.sda2.in‚128.625}at¤’…‚(:‡ ÚˆëÔ»é«â gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.048' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÿT;†‚(:° ÚˆëÔ놗å) disk./.usagePercent‚75.93129470356223 disk.sdb2.out‚823.5029296875 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.dm-0.out‚ 49678.703125 disk.sdb.out‚823.5029296875 disk.sda2.in‚128.625 disk.md0.out‚ 810.80078125) proc.memory.availableMB‚ 1316.31640625* proc.memory.percent‚18.227481842041016 network.recv‚ 1.08979e+06% disk./.usageGB‚2715.6708908081055 disk.sda2.out‚823.5029296875 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 810.80078125 disk.nvme0n1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sdb1.out‚0! proc.memory.rssMB‚ 2986.390625 cpu‚25.224789159697917 disk.sda.out‚823.5029296875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.out‚35202.5 disk.md0.in‚ 199.5859375$ memory_percent‚91.96584224700928! disk.nvme1n1.out‚ 14476.203125 disk.sda1.in‚0 network.sent‚ 5.674291e+06 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625Y¯·ô“‡‚(:ˆ éˆëÔˆøßà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍØGâBˆ‚(:· éˆëÔÿúà disk.nvme1n1.in‚0* proc.memory.percent‚18.227481842041016 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.sdb.out‚824.96435546875 disk.nvme0n1.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0p1.in‚ 199.5859375! proc.memory.rssMB‚ 2986.390625 network.sent‚ 5.682578e+06% disk./.usageGB‚2715.8896408081055 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 50044.1796875 disk.sda.in‚128.625 disk.rbd0.out‚1.125 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 812.1953125$ memory_percent‚91.96658134460449 cpu‚25.223782577984256 disk.sda2.in‚128.625! disk.sda2.out‚824.96435546875 disk.md0.out‚ 812.1953125( proc.memory.availableMB‚ 1316.1953125 disk.sda1.out‚0 disk.rbd2.out‚1.0625 disk.sda.out‚824.96435546875" disk.nvme1n1.out‚ 14476.3359375( disk./.usagePercent‚75.9374110451162 disk.nvme0n1.out‚ 35567.84375 network.recv‚ 1.094626e+06 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚824.96435546875íuyŸ”‰‚(:‰ øˆëÔó…¢à gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o»oGŠ‚(:¼ øˆëÔ«Íÿà) disk./.usagePercent‚75.94442962811142 disk.sdb2.in‚ 70.9609375 disk.sda.out‚826.71923828125! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 813.91015625# disk.nvme1n1.out‚14476.57421875) proc.memory.availableMB‚ 1316.42578125 cpu‚25.222714325859084 disk.sda1.in‚0 disk.sdb.out‚826.71923828125! disk.sdb2.out‚826.71923828125 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 813.91015625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 network.sent‚ 5.688679e+06% disk./.usageGB‚2716.1406593322754 disk.rbd2.out‚1.0625! disk.sda2.out‚826.71923828125 disk.dm-0.out‚ 50207.96875# memory_percent‚91.9651746749878 proc.cpu.threads‚17 network.recv‚ 1.099078e+06 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0# disk.nvme0n1.out‚35731.39453125 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.sda.in‚128.625_ÄHˆ2‹- _wandbruntime‚3128  _runtime‚3128QÆŒÀ & lr-AdamW/pg1‚0.0007027266673443678 & lr-AdamW/pg2‚0.0007027266673443678 trainer/global_step‚2599 $ _timestamp‚1.788527744578436e+09 _runtime‚3128.114040543 _step‚210È,ÏËÅ & lr-AdamW/pg1‚0.0007027266673443678 & lr-AdamW/pg2‚0.0007027266673443678 trainer/global_step‚2599 $ _timestamp‚1.788527744578436e+09 _runtime‚3128.114040543 _step‚210ÒÃ]ãW2Ž- _wandbruntime‚3128  _runtime‚3128/¥Wò'ìO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65838833 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32580862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244812861841408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977934394880 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65865176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚3128.115224349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54376768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32580860 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚211 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54376770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54376768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966798940555 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222049198593536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15849580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219070354227712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214788068623963 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65838833 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241697533780480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54376770 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 % _timestamp‚1.7885277445795913e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3115556843520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211821269683408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219070336224768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3115328060928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211821286722768 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65865176Ú㘰·ñO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15850208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244813114564096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65838833 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241697533780480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54376770 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 % _timestamp‚1.7885277445795913e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3115556843520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211821269683408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219070336224768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3115328060928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211821286722768 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65865176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65838833 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@Dev±9½@iceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32580862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244812861841408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977934394880 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65865176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚3128.115224349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54376768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32580860 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚211 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54376770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54376768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966798940555 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222049198593536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15849580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219070354227712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214788068623963 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3116718698496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2979055855104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65890033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222131011165696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2967916213833 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2979965839360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54396930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32592925 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7885277455311701e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2979055855104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚2599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219151955310592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244904502123520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48448824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211900304317312 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855897 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222131939152896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54396928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244904245219840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 _runtime‚3129.067282213 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222131939152896ïDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3116718698496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2979055855104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65890033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222131011165696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2967916213833 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2979965839360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54396930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32592925 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7885277455311701e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2979055855104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚2599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219151955310592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244904502123520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48448824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211900304317312 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855897 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222131939152896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353Ô¿¹2”- _wandbruntime‚3129  _runtime‚3129ØϳW•Ñ /loss/FBt3/detection_loss‚1.0383510627783834 trainer/global_step‚2599 _step‚213 2loss/SRqqt1/assignment_loss‚1.7045138097471662 1loss/SRqqt2/detection_loss‚2.7387097857215186 1loss/SRqqt3/detection_loss‚1.0627842173856847 0loss/SRqqt4/detection_loss‚1.817164532384094 %loss/total_loss‚45.79703020523785 0loss/FRt3/assignment_loss‚1.5336676144599914 1loss/FBt3/assignment_loss‚0.21724384570919583 .loss/FBt4/detection_loss‚2.289023826519648 2loss/SRqqt3/assignment_loss‚1.2031245324979811 /loss/FRt3/detection_loss‚1.6184564971923827 ,loss/FBt1/detection_loss‚2.3372732847929 .loss/FBt2/detection_loss‚2.374556714296341 % _timestamp‚1.7885277455335708e+09 .loss/FRt2/detection_loss‚2.517793544135094 0loss/FRt1/assignment_loss‚3.5712370586395266 0loss/FRt4/assignment_loss‚3.7908115916781955 2loss/SRqqt2/assignment_loss‚1.4544835964387113 1loss/FBt1/assignment_loss‚0.25369773510446975 1loss/FBt4/assignment_loss‚0.36656428799789054 /loss/FRt4/detection_loss‚2.2381180117527646 _runtime‚ 3129.06883877 0loss/FBt2/assignment_loss‚0.2421207955981572 .loss/FRt1/detection_loss‚2.992718176841736 0loss/SRqqt1/detection_loss‚2.453774772308491 epoch‚2 /loss/FRt2/assignment_loss‚4.497957364718119 2loss/SRqqt4/assignment_loss‚1.48288354653941129ƒqÑ\–Ö 2loss/SRqqt1/assignment_loss‚1.7045138097471662 1loss/SRqqt2/detection_loss‚2.7387097857215186 1loss/SRqqt3/detection_loss‚1.0627842173856847 0loss/SRqqt4/detection_loss‚1.817164532384094 %loss/total_loss‚45.79703020523785 0loss/FRt3/assignment_loss‚1.5336676144599914 1loss/FBt3/assignment_loss‚0.21724384570919583 .loss/FBt4/detection_loss‚2.289023826519648 2loss/SRqqt3/assignment_loss‚1.2031245324979811 /loss/FRt3/detection_loss‚1.6184564971923827 ,loss/FBt1/detection_loss‚2.3372732847929 .loss/FBt2/detection_loss‚2.374556714296341 % _timestamp‚1.7885277455335708e+09 .loss/FRt2/detection_loss‚2.517793544135094 0loss/FRt1/assignment_loss‚3.5712370586395266 0loss/FRt4/assignment_loss‚3.7908115916781955 2loss/SRqqt2/assignment_loss‚1.4544835964387113 1loss/FBt1/assignment_loss‚0.25369773510446975 1loss/FBt4/assignment_loss‚0.36656428799789054 /loss/FRt4/detection_loss‚2.2381180117527646 _runtime‚ 3129.06883877 0loss/FBt2/assignment_loss‚0.2421207955981572 .loss/FRt1/detection_loss‚2.992718176841736 0loss/SRqqt1/detection_loss‚2.453774772308491 epoch‚2 /loss/FRt2/assignment_loss‚4.497957364718119 2loss/SRqqt4/assignment_loss‚1.4828835465394112 /loss/FBt3/detection_loss‚1.0383510627783834 trainer/global_step‚2599 _step‚213Õ”BË’—‚(:‡ ‡‰ëÔßЄá gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*0íB˜‚(:· ‡‰ëÔ›·Úá# disk.nvme1n1.out‚14476.78515625 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 815.26953125 disk.sda1.out‚0 network.sent‚ 5.691865e+06) disk./.usagePercent‚75.91493927090615 disk.rbd2.out‚1.0625 disk.sdb.out‚828.1357421875 disk.sdb2.out‚828.1357421875 disk.rbd0.in‚0# disk.nvme0n1.out‚36010.88671875 disk.sda.out‚828.1357421875# memory_percent‚91.9651985168457' proc.memory.availableMB‚ 1316.421875% disk./.usageGB‚2715.0859413146973 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚1.125 disk.dm-0.out‚ 50487.671875 cpu‚25.221803090802414 disk.rbd2.in‚0 disk.sda2.out‚828.1357421875 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 network.recv‚ 1.103073e+06 disk.md0p1.out‚ 815.26953125 disk.nvme0n1.in‚0 disk.sda.in‚128.625* proc.memory.percent‚18.227481842041016 proc.cpu.threads‚17 disk.sdb1.in‚0! proc.memory.rssMB‚ 2986.3906250ÿ”™‚(:‰ –‰ëÔôüëá gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k@€Aš‚(:¶ –‰ëԬхä network.sent‚ 5.801394e+06 disk.sdb.in‚ 70.9609375 disk.sdb.out‚829.34619140625 disk.sda2.in‚128.625' proc.memory.availableMB‚ 1316.390625* proc.memory.percent‚18.227481842041016 disk.nvme1n1.out‚ 14477.0625 disk.sda.in‚128.625 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 36254.1171875 disk.md0.out‚ 816.41796875 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0( disk./.usagePercent‚75.9149291381556$ disk./.usageGB‚2715.085578918457 disk.dm-0.out‚ 50731.1796875 disk.sda.out‚829.34619140625 disk.sdb1.in‚0$ memory_percent‚91.96538925170898 cpu‚25.220794773475625 disk.dm-0.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚829.34619140625 disk.nvme1n1.in‚0 disk.sda1.out‚0! disk.sda2.out‚829.34619140625 network.recv‚ 1.109837e+06 disk.md0p1.out‚ 816.41796875 disk.md0.in‚ 199.5859375 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.sda1.in‚0çñ9H›‚(:½ ¥‰ëÔÁ‘Ãß% disk./.usageGB‚2714.4666709899902 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.rbd2.out‚1.0625 network.recv‚ 1.114401e+06# disk.nvme1n1.out‚14478.27734375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 817.91796875 disk.sda1.out‚0! disk.sda2.out‚830.90869140625$ memory_percent‚91.96467399597168 cpu‚25.219752678333624 network.sent‚ 5.805332e+06! disk.sdb2.out‚830.90869140625 disk.rbd0.out‚1.125( proc.memory.availableMB‚ 1316.5078125! proc.memory.rssMB‚ 2986.390625) disk./.usagePercent‚75.89762421344257 disk.dm-0.out‚ 50775.953125 disk.sdb1.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.in‚0* proc.memory.percent‚18.227481842041016 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sdb.out‚830.90869140625 disk.rbd0.in‚0# disk.nvme0n1.out‚36297.67578125 disk.md0.out‚ 817.91796875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sda.out‚830.90869140625 disk.rbd2.in‚0x+‰’œ‚(:‡ ¥‰ëÔÿʱà gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤n3”‚(:‰ ´‰ëÔƒæ“á gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 åÁ,ž‚(:¬ ´‰ëÔ² „â! proc.memory.rssEI,R MB‚ 2986.390625 cpu‚25.21883865300716 proc.cpu.threads‚17) disk./.usagePercent‚75.90156201362798 disk.dm-0.in‚0 disk.sdb.out‚832.6572265625 disk.nvme1n1.out‚ 14478.65625 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.sdb2.out‚832.6572265625 disk.sda2.in‚128.625 network.sent‚ 5.8115e+06 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.nvme0n1.in‚0$ memory_percent‚91.96619987487793 disk.rbd0.out‚1.125( proc.memory.availableMB‚ 1316.2578125* proc.memory.percent‚18.227481842041016 disk.sda.in‚128.625 disk.sda.out‚832.6572265625 disk.sdb1.in‚0 disk.md0.out‚ 819.60546875 network.recv‚ 1.118573e+06 disk.dm-0.out‚ 50848.5625 disk.sdb1.out‚0 disk.sda2.out‚832.6572265625 disk.md0p1.out‚ 819.60546875 disk.rbd0.in‚0# disk./.usageGB‚2714.60750579834 disk.rbd2.in‚0 disk.nvme0n1.out‚ 36369.90625|½ú2Ÿ- _wandbruntime‚3181  _runtime‚3181ÁڪƠÀ & lr-AdamW/pg1‚0.0007006282212570494 & lr-AdamW/pg2‚0.0007006282212570494 trainer/global_step‚2649 % _timestamp‚1.7885277978005483e+09 _runtime‚ 3181.33617711 _step‚214TÈ\’Ë¡Å & lr-AdamW/pg2‚0.0007006282212570494 trainer/global_step‚2649 % _timestamp‚1.7885277978005483e+09 _runtime‚ 3181.33617711 _step‚214 & lr-AdamW/pg1‚0.0007006282212570494Ö8@ö2¢- _wandbruntime‚3181  _runtime‚3181K*¶^ñ'£ëO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49321489 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67110078 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67083735 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223151290515968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223151308518912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215772149410608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034931790848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788527797802105e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218794825606271 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249382889890304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3174046947328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223151290515968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226185312335872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33184110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67110078 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3034021819904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3181.337959478 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55384768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3034021819904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55384768e/ç;ö'¤ðO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67110078 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67083735 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223151290515968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223151308518912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215772149410608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034931790848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788527797802105e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218794825606271 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249382889890304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3174046947328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223151290515968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226185312335872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33184110 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67110078 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3034021819904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3181.337959478 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55384768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3034021819904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55384768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49321489 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0׳PŠÚ2¥- _wandbruntime‚3182  _runtime‚3182ù¢Èü&¦öM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885277990754485e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122513494 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55404928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246299068647424 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223232927604736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218875885488661 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚3182.613165028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226268980869120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249474276545536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33196175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67134935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218874960474093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036053264384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3174974934528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33196177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036053264384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035143280128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3035143280128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215851184044512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226268980869120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684~f`_¥§ûM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3035143280128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215851184044512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226268980869120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885277990754485e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122513494 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55404928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246299068647424 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223232927604736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218875885488661 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚3182.613165028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226268980869120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249474276545536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33196175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67134935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218874960474093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036053264384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3174974934528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33196177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036053264384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035143280128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185Ø1¤Æ¤2¨- _wandbruntime‚3182  _runtime‚3182XÀ.Q©Ë 0loss/FRt2/assignment_loss‚3.5706816955848977 2loss/SRqqt3/assignment_loss‚1.2541522407044954 .loss/FBt1/detection_loss‚2.517120564977328 /loss/FBt3/detection_loss‚0.6823916401783073 1loss/FBt3/assignment_loss‚0.21218986756539213 0loss/SRqqt1/detection_loss‚2.555526075692012 .loss/FBt2/detection_loss‚2.548075495021684 .loss/FBt4/detection_loss‚2.172174023730414 trainer/global_step‚2649 2loss/SRqqt4/assignment_loss‚1.4242677187441655 .loss/FRt4/detection_loss‚2.372352375028034 1loss/SRqqt3/detection_loss‚0.8683887728557145 _runtime‚ 3182.61583428 .loss/FRt1/assignment_loss‚4.57573436607014 1loss/FBt2/assignment_loss‚0.16263141061920447 0loss/SRqqt4/detection_loss‚1.907264305173226 /loss/FRt3/assignment_loss‚1.193518045013898 /loss/FBt4/assignment_loss‚0.122545977903688 .loss/FRt3/detection_loss‚1.582568815003519 _step‚217 2loss/SRqqt2/assignment_loss‚1.7625239604350291 1loss/SRqqt2/detection_loss‚2.6677339645949276 /loss/FRt4/assignment_loss‚3.928381599485874 .loss/FRt2/detection_loss‚2.837781040756791 #loss/total_loss‚46.741071040345 epoch‚2 $ _timestamp‚1.788527799080406e+09 2loss/SRqqt1/assignment_loss‚2.2169491885037256 1loss/FBt1/assignment_loss‚0.29614381692022107 /loss/FRt1/detection_loss‚3.3099740797823127/·ûVªÐ 2loss/SRqqt1/assignment_loss‚2.2169491885037256 1loss/FBt1/assignment_loss‚0.29614381692022107 /loss/FRt1/detection_loss‚3.3099740797823127 0loss/FRt2/assignment_loss‚3.5706816955848977 2loss/SRqqt3/assignment_loss‚1.2541522407044954 .loss/FBt1/detection_loss‚2.517120564977328 /loss/FBt3/detection_loss‚0.6823916401783073 1loss/FBt3/assignment_loss‚0.21218986756539213 0loss/SRqqt1/detection_loss‚2.555526075692012 .loss/FBt2/detection_loss‚2.548075495021684 .loss/FBt4/detection_loss‚2.172174023730414 trainer/global_step‚2649 2loss/SRqqt4/assignment_loss‚1.4242677187441655 .loss/FRt4/detection_loss‚2.372352375028034 1loss/SRqqt3/detection_loss‚0.8683887728557145 _runtime‚ 3182.61583428 .loss/FRt1/assignment_loss‚4.57573436607014 1loss/FBt2/assignment_loss‚0.16263141061920447 0loss/SRqqt4/detection_loss‚1.907264305173226 /loss/FRt3/assignment_loss‚1.193518045013898 /loss/FBt4/assignment_loss‚0.122545977903688 .loss/FRt3/detection_loss‚1.582568815003519 _step‚217 2loss/SRqqt2/assignment_loss‚1.7625239604350291 1loss/SRqqt2/detection_loss‚2.6677339645949276 /loss/FRt4/assignment_loss‚3.928381599485874 .loss/FRt2/detection_loss‚2.837781040756791 #loss/total_loss‚46.741071040345 epoch‚2 $ _timestamp‚1.788527799080406e+09ÙójŠ;«‚(:° ÉëÔ÷²£à disk.rbd0.in‚0 disk.nvme0n1.out‚36514.25 disk.dm-0.out‚ 50993.1875! disk.sdb2.out‚834.25634765625 network.recv‚ 1.121712e+06% disk./.usageGB‚2714.6388816833496 disk.sdb1.out‚0 disk.sda2.in‚128.625) proc.memory.availableMB‚ 1316.48046875 disk.sdb1.in‚0 disk.md0.out‚ 821.1484375 network.sent‚ 5.816714e+06) disk./.usagePercent‚75.90243929650468 disk.rbd0.out‚1.125 disk.sda1.out‚0 disk.dm-0.in‚0 disk.md0.in‚ 199.5859375 disk.nvme1n1.out‚ 14478.9375$ memory_percent‚91.96484088897705* proc.memory.percent‚18.227481842041016 disk.sda.out‚834.25634765625 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚834.25634765625 cpu‚25.217770138038468 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.nvme0n1.in‚0 disk.rbd2.in‚0! disk.sda2.out‚834.25634765625 disk.md0p1.out‚ 821.1484375}Óu“¬‚(:ˆ ÉëÔÀÙÙà gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09‘Oõ”­‚(:‰ Ò‰ëÔ×ý´à gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%*U’E®‚(:º Ò‰ëÔµ†›â disk.nvme0n1.in‚0 disk.sda2.in‚128.625# disk.nvme1n1.out‚14479.37109375 disk.md0p1.in‚ 199.5859375( proc.memory.availableMB‚ 658.36328125! disk.sda2.out‚835.45751953125 disk.md0.in‚ 199.5859375 disk.sdb1.in‚0 disk.rbd2.out‚1.0625$ memory_percent‚95.98166942596436* proc.memory.percent‚18.227481842041016 disk.sdb1.out‚0 disk.sda1.in‚0 disk.dm-0.out‚51094.26953125% disk./.usageGB‚2714.8287391662598) disk./.usagePercent‚75.90774779118837 disk.sdb.out‚835.45751953125 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0! disk.sdb2.out‚835.45751953125 disk.sda1.out‚0 disk.rbd2.in‚0 cpu‚25.216805804939217 disk.md0p1.out‚ 822.2734375 disk.sdb.in‚ 70.9609375 network.sent‚ 5.925298e+06 network.recv‚ 1.1282e+06" disk.nvme0n1.out‚ 36614.8984375 disk.md0.out‚ 822.2734375 disk.dm-0.in‚0 disk.sda.out‚835.45751953125 disk.rbd0.out‚1.125! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17“⸔¯‚(:‰ á‰ëÔ­èâà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a/ .°‚(:£ á‰ëÔ–çÚá! disk.nvme0n1.out‚ 36860.609375 disk.sda2.in‚128.625 disk.md0.out‚ 823.86328125 disk.sdb.out‚ 837.09375 disk.md0p1.out‚ 823.86328125* proc.memory.percent‚18.227481842041016 network.sent‚ 5.931465e+06 disk.dm-0.out‚51340.13671875 disk.sda.out‚ 837.09375# disk.nvme1n1.out‚14479.52734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚ 837.09375 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚1.125 disk.sdb1.out‚0) disk./.usagePercent‚75.92348437943708 disk.md0.in‚ 199.5859375 disk.sdb2.out‚ 837.09375$ memory_percent‚95.97938060760498 network.recv‚ 1.133286e+06$ disk./.usageGB‚2715.391555786133 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 658.73828125 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.nvme1n1.in‚0! proc.memory.rssMB‚ 2986.390625 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 cpu‚25.21596683250053 disk.sdb2.in‚ 70.9609375&2 #2±- _wandbruntime‚3238  _runtime‚3238Q›uPDzÁ % _timestamp‚1.7885278553942153e+09 _runtime‚3238.929759523 _step‚218 & lr-AdamW/pg1‚0.0006984600182554116 & lr-AdamW/pg2‚0.0006984600182554116 trainer/global_step‚2699ÖwQtÌ³Æ _step‚218 & lr-AdamW/pg1‚0.0006984600182554116 & lr-AdamW/pg2‚0.0006984600182554116 trainer/global_step‚2699 % _timestamp‚1.7885278553942153e+09 _runtime‚3238.929759523Úº«‹ü2´- _wandbruntime‚3238  _runtime‚3238Õ4eiñ'µëO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253952505180160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227232244807168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50213812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50213175 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 $ _timestamp‚1.788527855394949e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250720128165376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3238.930580287 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3232135649280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091019215872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3090109244928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56392768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68354980 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68328637 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16425815 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253952239874560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250720104225280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230323282025984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615}]Èû ¶ðO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230323282025984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253952505180160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227232244807168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50213812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50213175 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 $ _timestamp‚1.788527855394949e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250720128165376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3238.930580287 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3232135649280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091019215872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3090109244928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56392768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68354980 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68328637 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16425815 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253952239874560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250720104225280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3240.110850114 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68379837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227313863892992 _step‚220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3233303854080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222882642470969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227313863892992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56412928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16431393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254043888590336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 $ _timestamp‚1.788527856573367e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227313881895936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092140689408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50231463 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219802063771712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚11857—Ym'»úM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33799427 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68379837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227313863892992 _step‚220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3233303854080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222882642470969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227313863892992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56412928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16431393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254043888590336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 $ _timestamp‚1.788527856573367e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227313881895936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092140689408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50231463 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219802063771712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3240.110850114 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 F¨ôñc¾Ý /loss/FRt4/assignment_loss‚2.955319960912069 epoch‚2 1loss/FBt2/assignment_loss‚0.19956636068689643 0loss/FBt4/assignment_loss‚0.7455915638129227 1loss/SRqqt3/detection_loss‚0.9191663865709588 1loss/FBt3/assignment_loss‚0.21356960975165942 .loss/FBt1/detection_loss‚2.560565022321848 /loss/FBt2/detection_loss‚2.1372102228077976 .loss/FRt2/detection_loss‚2.538959143623229 2loss/SRqqt2/assignment_loss‚1.8463049728423357 2loss/SRqqt3/assignment_loss‚1.0989001206388431 /loss/FRt1/detection_loss‚2.8563149499170706 1loss/SRqqt4/detection_loss‚1.6062589808115884 _step‚221 0loss/FRt1/assignment_loss‚3.8902902856017603 /loss/FRt3/detection_loss‚1.7833016017382894 0loss/SRqqt2/detection_loss‚2.067890353500843 trainer/global_step‚2699 0loss/FRt3/assignment_loss‚1.4599037026917492 1loss/FBt1/assignment_loss‚0.05754053967879932 0loss/FBt3/detection_loss‚0.48510161918443695 _runtime‚3240.112268781 0loss/FRt2/assignment_loss‚3.8237038581602034 2loss/SRqqt1/assignment_loss‚2.0580341913737357 .loss/FBt4/detection_loss‚4.022087723016739 %loss/total_loss‚45.80397010550841 % _timestamp‚1.7885278565767007e+09 2loss/SRqqt4/assignment_loss‚1.7746379763361007 /loss/FRt4/detection_loss‚2.2856839056368226 1loss/SRqqt1/detection_loss‚2.4180670538917184Ý*¦Ÿ”¿‚(:‰ ÿ‰ëÔÅÈ´à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢­ FÀ‚(:» ÿ‰ëÔ»¨úá disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14479.98046875 cpu‚25.21421549986798 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.sda.out‚839.88916015625 disk.md0p1.out‚ 826.52734375 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 6.044147e+06! disk.sda2.out‚839.88916015625 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0) disk./.usagePercent‚75.93440076491814 disk.sda2.in‚128.625' proc.memory.availableMB‚ 658.4453125% disk./.usageGB‚2715.7819786071777 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb2.out‚839.88916015625 disk.sdb.out‚839.88916015625* proc.memory.percent‚18.227481842041016# disk.nvme0n1.out‚37329.58984375 disk.md0.out‚ 826.52734375 disk.dm-0.out‚ 51809.5703125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2986.390625 network.recv‚ 1.14396e+06 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚1.125$ memory_percent‚95.98116874694824‚ÿQo’Á‚(:‡ ŽŠëÔÜÃÑá gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ï©‚7‚(:¬ ŽŠëÔ…ÇÖã network.recv‚ 1.149831e+06 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚95.98417282104492* proc.memory.percent‚18.227481842041016 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 841.525390625 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.md0.out‚ 828.109375 disk.dm-0.out‚52122.12890625! proc.memory.rssMB‚ 2986.390625% disk./.usageGB‚2715.7819786071777 disk.sdb2.out‚ 841.525390625 disk.sda.in‚128.625 disk.sda.out‚ 841.525390625 disk.sdb1.out‚0 disk.dm-0.in‚0 network.sent‚ 6.130372e+06 disk.md0p1.out‚ 828.109375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 841.525390625) disk./.usagePercent‚75.93440076491814 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375& proc.memory.availableMB‚ 657.953125 disk.sda2.in‚128.625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14480.19140625 disk.sda1.in‚0 disk.nvme0n1.out‚ 37641.9375 cpu‚25.21328831301092 »( “Â(:ˆ ŠëÔÝÁ¤à gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uª…}Ä‚(:² ŠëÔ¼¬Ùá! disk.sdb2.out‚842.99951171875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚128.6Þ¾À25 disk.md0.out‚ 829.53125 disk.sda.out‚842.99951171875% proc.memory.availableMB‚ 658.28125 network.recv‚ 1.154255e+06 disk.sda1.in‚0 cpu‚25.212353001677922( disk./.usagePercent‚75.9196576128656 disk.sdb.out‚842.99951171875 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 disk.rbd0.out‚1.125 disk.sda.in‚128.625! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17% disk./.usageGB‚2715.2546920776367 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 network.sent‚ 6.136526e+06 disk.dm-0.in‚0 disk.md0p1.out‚ 829.53125$ memory_percent‚95.98217010498047! disk.sda2.out‚842.99951171875# disk.nvme1n1.out‚14480.30859375 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 37876.65625 disk.dm-0.out‚52356.96484375 disk.sdb.in‚ 70.9609375É7ý2Å- _wandbruntime‚3295  _runtime‚3295?RÆÆÀ & lr-AdamW/pg1‚0.0006962225101897107 & lr-AdamW/pg2‚0.0006962225101897107 trainer/global_step‚2749 $ _timestamp‚1.788527911567525e+09 _runtime‚3295.103187828 _step‚222u¦tŸËÇÅ & lr-AdamW/pg1‚0.0006962225101897107 & lr-AdamW/pg2‚0.0006962225101897107 trainer/global_step‚2749 $ _timestamp‚1.788527911567525e+09 _runtime‚3295.103187828 _step‚222ÞKó{ƒ2È- _wandbruntime‚3295  _runtime‚3295¬5ò'ÉìO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57400770 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223673925904368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231313217101312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3147106640896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3146196669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3146196669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255231389447680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885279115687103e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234460323742208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223673908865008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3134430705879 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16713404 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34390610 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258522205188096ÕÚž÷'ÊñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231313199098368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234459395768320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258521954562560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57400770 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104636 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226809264585247 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231313199098368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3290791800320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚2749 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57400768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3290565114880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3134430705879 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16713404 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34390610 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258522205188096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57400770 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223673925904368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231313217101312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3147106640896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3146196669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3146196669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255231389447680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885279115687103e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234460323742208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223673908865008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231394836187136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3148228114432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234543064301568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3136455954365 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7885279125353549e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231394836187136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57420930 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69598370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258613605271040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223752926459552 trainer/global_step‚2749 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69624739 _runtime‚3296.071660169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _step‚224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127019298–ǹ™îÍûM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.cur‚B?ñ$rent‚ 909984256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234543064301568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3136455954365 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7885279125353549e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231394836187136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57420930 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69598370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258613605271040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223752926459552 trainer/global_step‚2749 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69624739 _runtime‚3296.071660169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _step‚224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127019298 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234543064301568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231394818184192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231394836187136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3148228114432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185àÓé 2Î- _wandbruntime‚3296  _runtime‚3296$ú*Ó\ÏÖ 0loss/FRt1/assignment_loss‚3.5810390843285456 0loss/FBt3/assignment_loss‚0.2518222701835537 1loss/SRqqt1/detection_loss‚2.4397446980530564 1loss/SRqqt4/detection_loss‚1.7349739712426218 % _timestamp‚1.7885279125373898e+09 /loss/FRt2/assignment_loss‚4.301710985208812 0loss/FRt4/assignment_loss‚3.2148489120759463 1loss/SRqqt2/assignment_loss‚2.316388951712533 1loss/FBt4/assignment_loss‚0.05557514177635312 /loss/FBt1/detection_loss‚3.5146148800849915 /loss/FBt3/detection_loss‚0.8136338220126387 epoch‚2 2loss/SRqqt3/assignment_loss‚1.0333235146576636 2loss/SRqqt4/assignment_loss‚1.8378547044723468 /loss/FRt3/detection_loss‚2.0172310622279035 1loss/FBt1/assignment_loss‚0.02249624556861818 trainer/global_step‚2749 2loss/SRqqt1/assignment_loss‚2.5767347021875056 2loss/FBt2/assignment_loss‚0.013343486188205134 .loss/FRt1/detection_loss‚2.856899400552114 .loss/FBt2/detection_loss‚4.869349732995033 %loss/total_loss‚50.06886452203952 0loss/FRt3/assignment_loss‚1.4139306101688118 .loss/FRt2/detection_loss‚2.427358912795572 .loss/FRt4/detection_loss‚2.507699403911829 1loss/SRqqt2/detection_loss‚2.2020150862242045 .loss/FBt4/detection_loss‚3.073823893070221 _runtime‚3296.072852425 _step‚225 1loss/SRqqt3/detection_loss‚0.9924510503404362IF9}aÐÛ % _timestamp‚1.7885279125373898e+09 /loss/FRt2/assignment_loss‚4.301710985208812 0loss/FRt4/assignment_loss‚3.2148489120759463 1loss/SRqqt2/assignment_loss‚2.316388951712533 1loss/FBt4/assignment_loss‚0.05557514177635312 /loss/FBt1/detection_loss‚3.5146148800849915 /loss/FBt3/detection_loss‚0.8136338220126387 epoch‚2 2loss/SRqqt3/assignment_loss‚1.0333235146576636 2loss/SRqqt4/assignment_loss‚1.8378547044723468 /loss/FRt3/detection_loss‚2.0172310622279035 1loss/FBt1/assignment_loss‚0.02249624556861818 trainer/global_step‚2749 2loss/SRqqt1/assignment_loss‚2.5767347021875056 2loss/FBt2/assignment_loss‚0.013343486188205134 .loss/FRt1/detection_loss‚2.856899400552114 .loss/FBt2/detection_loss‚4.869349732995033 %loss/total_loss‚50.06886452203952 0loss/FRt3/assignment_loss‚1.4139306101688118 .loss/FRt2/detection_loss‚2.427358912795572 .loss/FRt4/detection_loss‚2.507699403911829 1loss/SRqqt2/detection_loss‚2.2020150862242045 .loss/FBt4/detection_loss‚3.073823893070221 _runtime‚3296.072852425 _step‚225 1loss/SRqqt3/detection_loss‚0.9924510503404362 0loss/FRt1/assignment_loss‚3.5810390843285456 0loss/FBt3/assignment_loss‚0.2518222701835537 1loss/SRqqt1/detection_loss‚2.4397446980530564 1loss/SRqqt4/detection_loss‚1.7349739712426218á)¬k”Ñ‚(:‰ ¬ŠëÔˆ½¯á gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mSÃ3Ò‚(:¨ ¬ŠëÔªêóá disk.sdb1.in‚0 disk.dm-0.out‚52401.59765625 disk.sda.in‚128.625 disk.sda.out‚ 844.4765625 disk.nvme0n1.in‚0* proc.memory.percent‚18.227481842041016 cpu‚25.2114667993057 disk.rbd2.in‚0 disk.sda2.out‚ 844.4765625 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚1.125' proc.memory.availableMB‚ 658.4609375! proc.memory.rssMB‚ 2986.390625 disk.rbd2.out‚1.0625# memory_percent‚95.9810733795166 proc.cpu.threads‚17 network.recv‚ 1.158905e+06) disk./.usagePercent‚75.91938637513242 disk.sdb.out‚ 844.4765625 disk.sdb2.out‚ 844.4765625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 830.9453125 disk.md0.in‚ 199.5859375 disk.md0.out‚ 830.9453125# disk.nvme0n1.out‚37920.93359375 network.sent‚ 6.140511e+06 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 14480.6640625 disk.rbd0.in‚0 disk.sda1.in‚0% disk./.usageGB‚2715.2449913024902 disk.sdb2.in‚ 70.9609375î²Ä3”Ó‚(:‰ »ŠëÔ–ªžà gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{0i!@Ô‚(:µ »ŠëÔû‡Æá disk.dm-0.out‚ 52685.4921875! disk.sdb2.out‚845.89404296875 disk.sda1.in‚0" disk.nvme0n1.out‚ 38204.6171875& proc.memory.availableMB‚ 658.453125 proc.cpu.threads‚17) disk./.usagePercent‚75.92042204889933 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.227481842041016 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚0! proc.memory.rssMB‚ 2986.390625 cpu‚25.21052249637045 disk.rbd0.in‚0% disk./.usageGB‚2715.2820320129395 disk.sda.in‚128.625 disk.nvme1n1.in‚0 network.sent‚ 6.249615e+06 network.recv‚ 1.165276e+06 disk.sda.out‚845.89404296875 disk.rbd0.out‚1.125 disk.rbd2.out‚1.0625! disk.sda2.out‚845.89404296875 disk.md0.in‚ 199.5859375 disk.md0p1.out‚ 832.29296875 disk.nvme1n1.out‚ 14480.875 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.md0.out‚ 832.29296875 disk.sdb.out‚845.89404296875$ memory_percent‚95.98112106323242¥]ãW’Õ‚(:‡ ÊŠëÔ娓à gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽLOcEÖ‚(:º ÊŠëÔÁÝà) disk./.usagePercent‚75.92391528798686 disk.sda.in‚128.625# disk.nvme1n1.out‚14481.28515625 disk.md0p1.out‚ 833.79296875 disk.sdb1.out‚0 disk.dm-0.in‚0! proc.memory.rssMB‚ 2986.390625 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0 disk.md0.out‚ 833.79296875' proc.memory.availableMB‚ 657.9609375 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 network.sent‚ 6.255236e+06$ disk./.usageGB‚2715.406967163086! disk.sdb2.out‚847.43798828125 disk.sda.out‚847.43798828125 disk.nvme0n1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.169004e+06! disk.sda2.out‚847.43798828125 disk.sda1.out‚0 disk.rbd2.out‚1.0625* proc.memory.percent‚18.227481842041016 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sda2.in‚128.625 disk.dm-0.out‚ 52973.3984375# memory_percent‚95.9841251373291 cpu‚25.20981723815679 disk.sdb.out‚847.43798828125 disk.rbd0.in‚0 disk.rbd0.out‚1.125# disk.nvme0n1.out‚38492.11328125‰8ÿ”ׂ(:‰ ÙŠëÔ°ä§à gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.44425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šäÄ?Ø‚(:´ ÙŠëÔ×Àöà) disk./.usagePercent‚75.92741023364287# disk.nvme0n1.out‚38721.37890625 disk.md0.out‚ 835.18359375 disk.sda.out‚ 848.873046875( proc.memory.availableMB‚ 658.45703125% disk./.usageGB‚2715.5319633483887 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 835.18359375 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2986.390625 network.sent‚ 6.261337e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 53202.84375 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0 disk.sdb2.out‚ 848.873046875* proc.memory.percent‚18.227481842041016 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚ 848.873046875 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14481.46484375 disk.md0p1.in‚ 199.5859375 network.recv‚ 1.173642e+06 disk.sdb.out‚ 848.873046875 disk.sda.in‚128.625$ memory_percent‚95.98109722137451 cpu‚25.208954211999703#èf2Ù- _wandbruntime‚3352  _runtime‚33526ƳÇÚÁ % _timestamp‚1.7885279686303794e+09 _runtime‚3352.166039075 _step‚226 & lr-AdamW/pg1‚0.0006939161633532738 & lr-AdamW/pg2‚0.0006939161633532738 trainer/global_step‚2799K§A¢ÌÛÆ % _timestamp‚1.7885279686303794e+09 _runtime‚3352.166039075 _step‚226 & lr-AdamW/pg1‚0.0006939161633532738 & lr-AdamW/pg2‚0.0006939161633532738 trainer/global_step‚2799â¿æC¹2Ü- _wandbruntime‚3352  _runtime‚3352®¢gÇò'ÝìO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230816021567555 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58408770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3348929477120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235394153389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227624805631568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238597365458432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227624788592208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235394171392512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51995657 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58408770 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203194065920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17002418 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885279686317742e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58408770 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203194065920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17002418 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885279686317742e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58408770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3348929477120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235394153389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227624805631568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238597365458432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227624788592208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235394171392512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51995657 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58428928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192333209473 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52013987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235475790478336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚228 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3203405555200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3353.341957639 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204315539456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70869641 $ _timestamp‚1.788527969804484e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227703823226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58428930 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0*ðA0'áúM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58428930 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129298571 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235475772475392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230895231421017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58428928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192333209473 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52013987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235475790478336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚228 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3203405555200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3353.341957639 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204315539456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70869641 $ _timestamp‚1.788527969804484e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227703823226112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ä_ Gí2â- _wandbruntime‚3353  _runtime‚3353²CL®]ã× % _timestamp‚1.7885279698085296e+09 2loss/SRqqt2/assignment_loss‚1.9201020610332489 .loss/FBt4/detection_loss‚2.380902223289013 1loss/FBt3/assignment_loss‚0.18630883047748548 1loss/FBt4/assignment_loss‚0.08468008269301208 /loss/FRt1/detection_loss‚3.0410591769218445 1loss/SRqqt4/detection_loss‚2.0698137934270657 epoch‚2 _step‚229 /loss/FRt1/assignment_loss‚4.116178460121155 /loss/FRt2/assignment_loss‚3.293123652195108 0loss/FRt4/assignment_loss‚3.4459101822641163 2loss/SRqqt3/assignment_loss‚0.8333899017423392 /loss/FRt3/detection_loss‚1.6038945906541564 1loss/SRqqt2/detection_loss‚2.5725353860855105 1loss/SRqqt3/detection_loss‚0.9089414723217487 /loss/FBt1/detection_loss‚1.7260076009801455 1loss/FBt2/assignment_loss‚0.38068315394936386 /loss/FBt2/detection_loss‚2.5219485759735107 &loss/total_loss‚44.659203162979324 _runtime‚3353.344082979 0loss/FRt3/assignment_loss‚1.5891510302370244 -loss/FRt2/detection_loss‚2.58819050007853 1loss/SRqqt1/detection_loss‚2.4128249624501104 /loss/FBt3/detection_loss‚0.6883579165679058 2loss/SRqqt4/assignment_loss‚1.9218463419299376 0loss/FBt1/assignment_loss‚0.1277190695241838 trainer/global_step‚2799 2loss/SRqqt1/assignment_loss‚1.6033936423128066 .loss/FRt4/detection_loss‚2.642240555749999á¾dbäÜ 1loss/FBt3/assignment_loss‚0.18630883047748548 1loss/FBt4/assignment_loss‚0.08468008269301208 /loss/FRt1/detection_loss‚3.0410591769218445 1loss/SRqqt4/detection_loss‚2.0698137934270657 epoch‚2 _step‚229 /loss/FRt1/assignment_loss‚4.116178460121155 /loss/FRt2/assignment_loss‚3.293123652195108 0loss/FRt4/assignment_loss‚3.4459101822641163 2loss/SRqqt3/assignment_loss‚0.8333899017423392 /loss/FRt3/detection_loss‚1.6038945906541564 1loss/SRqqt2/detection_loss‚2.5725353860855105 1loss/SRqqt3/detection_loss‚0.9089414723217487 /loss/FBt1/detection_loss‚1.7260076009801455 1loss/FBt2/assignment_loss‚0.38068315394936386 /loss/FBt2/detection_loss‚2.5219485759735107 &loss/total_loss‚44.659203162979324 _runtime‚3353.344082979 0loss/FRt3/assignment_loss‚1.5891510302370244 -loss/FRt2/detection_loss‚2.58819050007853 1loss/SRqqt1/detection_loss‚2.4128249624501104 /loss/FBt3/detection_loss‚0.6883579165679058 2loss/SRqqt4/assignment_loss‚1.9218463419299376 0loss/FBt1/assignment_loss‚0.1277190695241838 trainer/global_step‚2799 2loss/SRqqt1/assignment_loss‚1.6033936423128066 .loss/FRt4/detection_loss‚2.642240555749999 % _timestamp‚1.7885279698085296e+09 2loss/SRqqt2/assignment_loss‚1.9201020610332489 .loss/FBt4/detection_loss‚2.380902223289013åLÕ¾{”å‚(:‰ èŠëÔÿÁ‘á gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¯ÈìEæ‚(:º èŠëÔᨭá disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 836.60546875 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 53391.1015625* proc.memory.percent‚18.227481842041016( disk./.usagePercent‚75.9344009782392 disk.nvme0n1.in‚0 disk.md0.out‚ 836.60546875 disk.dm-0.in‚0 disk.sdb.out‚850.34326171875! disk.sdb2.out‚850.34326171875 disk.sda.out‚850.34326171875! proc.memory.rssMB‚ 2986.390625% disk./.usageGB‚2715.7819862365723! disk.sda2.out‚850.34326171875 disk.sda.in‚128.625& proc.memory.availableMB‚ 658.140625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb2.in‚ 70.9609375 network.sent‚ 6.27837e+06 disk.sda1.out‚0 disk.rbd2.out‚1.0625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14482.28515625 network.recv‚ 1.178064e+06 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375$ memory_percent‚95.98302841186523 cpu‚25.208095287269337# disk.nvme0n1.out‚38908.81640625¯n¾Hç‚(:½ ÷ŠëÔ›ë÷ß disk.sdb.out‚852.33935546875 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 838.52734375 disk.nvme0n1.in‚0$ disk./.usageGB‚2715.782012939453 disk.sda.in‚128.625# disk.nvme1n1.out‚14482.47265625" disk.nvme0n1.out‚ 39220.9765625 network.recv‚ 1.184569e+06 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375( proc.memory.availableMB‚ 657.90234375) disk./.usagePercent‚75.93440172486294 disk.dm-0.out‚53703.44921875 cpu‚25.207346047010954 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda2.in‚128.625! disk.sda2.out‚852.33935546875$ memory_percent‚95.98448276519775 network.sent‚ 6.372488e+06 disk.md0p1.in‚ 199.5859375! disk.sdb2.out‚852.33935546875 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.md0.out‚ 838.52734375 disk.dm-0.in‚0! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 disk.sda.out‚852.33935546875 disk.rbd0.out‚1.125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375&¨8 ’è‚(:‡ ÷ŠëÔ–½Ìà gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚010°”é‚(:‰ †‹ëÔÔ—ßÞ gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W¼Ó›ê‚(:¦ †‹ëÔÚ¹¡à proc.cpu.threads‚17 disk.dm-0.out‚53956.93359375 disk.sdb2.out‚ 853.619140625 disk.sdb1.in‚0 disk.dm-0.in‚@Oç–0 network.sent‚ 6.38048e+06 disk.sda1.in‚0* proc.memory.percent‚18.227481842041016 disk.sda.in‚128.625 disk.sda.out‚ 853.619140625 disk.rbd2.out‚1.0625 network.recv‚ 1.188572e+06$ disk./.usageGB‚2715.254421234131 disk.sda2.out‚ 853.619140625& proc.memory.availableMB‚ 657.640625 disk.sda2.in‚128.625 disk.sdb.out‚ 853.619140625" disk.nvme1n1.out‚ 14482.6953125 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚839.75! proc.memory.rssMB‚ 2986.390625 disk.nvme0n1.in‚0# disk.nvme0n1.out‚39474.23828125 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.out‚839.75 disk.sdb.in‚ 70.9609375 cpu‚25.206576626546312) disk./.usagePercent‚75.91965003996783 disk.nvme1n1.in‚0$ memory_percent‚95.98608016967773 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375»x5n”ë‚(:‰ •‹ëÔ…é—ß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|kW¸=ì‚(:² •‹ëԢĴà disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375) disk./.usagePercent‚75.91840211174201$ disk./.usageGB‚2715.209789276123 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sda.out‚ 855.087890625$ memory_percent‚95.98329067230225* proc.memory.percent‚18.227481842041016 network.recv‚ 1.193028e+06 disk.rbd0.in‚0 disk.md0.out‚ 841.1796875 disk.dm-0.out‚54114.87109375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.out‚1.0625 disk.md0p1.out‚ 841.1796875 proc.cpu.threads‚17 disk.rbd0.out‚1.125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚0# disk.nvme1n1.out‚14482.88671875 network.sent‚ 6.386606e+06 disk.sda1.in‚0 disk.sda2.out‚ 855.087890625 disk.sdb2.out‚ 855.087890625( proc.memory.availableMB‚ 658.09765625 disk.nvme0n1.in‚0! proc.memory.rssMB‚ 2986.390625! disk.nvme0n1.out‚ 39631.984375 disk.md0.in‚ 199.5859375 disk.sdb.out‚ 855.087890625 cpu‚25.205743350706143?²Èq2í- _wandbruntime‚3409  _runtime‚3409ƒ{ö¤ÆîÀ & lr-AdamW/pg2‚0.0006915414583853245 trainer/global_step‚2849 $ _timestamp‚1.788528025850577e+09 _runtime‚3409.386128271 _step‚230 & lr-AdamW/pg1‚0.0006915414583853245Ž˜ž¯ËïÅ _runtime‚3409.386128271 _step‚230 & lr-AdamW/pg1‚0.0006915414583853245 & lr-AdamW/pg2‚0.0006915414583853245 trainer/global_step‚2849 $ _timestamp‚1.788528025850577e+09æÀš;—2ð- _wandbruntime‚3409  _runtime‚3409ÃîÄ=ò'ñìO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59416770 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72063343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35597112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239475107680768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 _step‚231 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242733479200768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258371520000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234822778549863 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 trainer/global_step‚2849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234821853535503 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72063343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239475125683712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3407322808832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3246185216095 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3407102414848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72089686 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72089686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3409.387316237 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0’ߌ½÷'òñO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 trainer/global_step‚2849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234821853535503 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72063343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239475125683712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3407322808832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3246185216095 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3407102414848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72089686 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72089686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3409.387316237 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59416770 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72063343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35597112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239475107680768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 _step‚231 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242733479200768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258371520000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234822778549863 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095ç; Ú2ó- _wandbruntime‚3410  _runtime‚3410:ûù9<$ôõM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 trainer/global_step‚2849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259492980224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3408244171776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267752429768704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259492980224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59436928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocä½J¿ation.small_pool.freed‚72088174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3410.501806269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816•›í'õúM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72088174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3410.501806269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 trainer/global_step‚2849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259492980224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3408244171776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267752429768704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259492980224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59436928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725觓¨2ö- _wandbruntime‚3410  _runtime‚3410{ó[÷Õ 1loss/SRqqt3/detection_loss‚1.0441880823628416 epoch‚2 trainer/global_step‚2849 0loss/FRt3/assignment_loss‚1.4972951779531878 /loss/FRt3/detection_loss‚1.2674953408192757 0loss/SRqqt2/detection_loss‚2.266234948299825 /loss/FBt1/detection_loss‚1.9645261822208282 &loss/total_loss‚45.413426843440675 1loss/FBt4/assignment_loss‚0.21673633734965397 1loss/SRqqt1/detection_loss‚2.7127590303619704 /loss/FBt2/detection_loss‚2.5582607739111958 /loss/FBt3/detection_loss‚0.8620850749846015 /loss/FBt4/detection_loss‚2.4749009891560205 /loss/FRt2/assignment_loss‚4.217264783879121 0loss/FRt4/assignment_loss‚3.5864994089777875 2loss/SRqqt2/assignment_loss‚1.3093548435717821 2loss/SRqqt4/assignment_loss‚2.0284285831909914 0loss/FBt1/assignment_loss‚0.3231082824796402 1loss/FBt2/assignment_loss‚0.22173788494356975 .loss/FRt4/detection_loss‚2.595325728742088 .loss/FRt1/assignment_loss‚2.95867782831192 % _timestamp‚1.7885280269685414e+09 /loss/FRt1/detection_loss‚3.0413382135588547 2loss/SRqqt3/assignment_loss‚1.1032128924757099 0loss/SRqqt4/detection_loss‚2.737861385712257 _runtime‚3410.503919238 1loss/SRqqt1/assignment_loss‚1.413639958947897 1loss/FBt3/assignment_loss‚0.18102121929589096 .loss/FRt2/detection_loss‚2.831473891933759 _step‚233W[ly`øÚ 0loss/SRqqt4/detection_loss‚2.737861385712257 _runtime‚3410.503919238 1loss/SRqqt1/assignment_loss‚1.413639958947897 1loss/FBt3/assignment_loss‚0.18102121929589096 .loss/FRt2/detection_loss‚2.831473891933759 _step‚233 1loss/SRqqt3/detection_loss‚1.0441880823628416 epoch‚2 trainer/global_step‚2849 0loss/FRt3/assignment_loss‚1.4972951779531878 /loss/FRt3/detection_loss‚1.2674953408192757 0loss/SRqqt2/detection_loss‚2.266234948299825 /loss/FBt1/detection_loss‚1.9645261822208282 &loss/total_loss‚45.413426843440675 1loss/FBt4/assignment_loss‚0.21673633734965397 1loss/SRqqt1/detection_loss‚2.7127590303619704 /loss/FBt2/detection_loss‚2.5582607739111958 /loss/FBt3/detection_loss‚0.8620850749846015 /loss/FBt4/detection_loss‚2.4749009891560205 /loss/FRt2/assignment_loss‚4.217264783879121 0loss/FRt4/assignment_loss‚3.5864994089777875 2loss/SRqqt2/assignment_loss‚1.3093548435717821 2loss/SRqqt4/assignment_loss‚2.0284285831909914 0loss/FBt1/assignment_loss‚0.3231082824796402 1loss/FBt2/assignment_loss‚0.22173788494356975 .loss/FRt4/detection_loss‚2.595325728742088 .loss/FRt1/assignment_loss‚2.95867782831192 % _timestamp‚1.7885280269685414e+09 /loss/FRt1/detection_loss‚3.0413382135588547 2loss/SRqqt3/assignment_loss‚1.1032128924757099éÐ$¬û2ù‚(:§ ¤‹ëÔŸ”Íà disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.rbd2.out‚1.0625& proc.memory.availableMB‚ 657.828125) disk./.usagePercent‚75.92042364880731 disk.dm-0.out‚ 54315.46875 disk.md0p1.out‚ 842.67578125 disk.rbd0.out‚1.125 proc.cpu.threads‚17% disk./.usageGB‚2715.2820892333984 disk.sdb.out‚ 856.65234375 disk.md0p1.in‚ 199.5859375 disk.sda2.out‚ 856.65234375* proc.memory.percent‚18.227481842041016 disk.sda.in‚128.625 disk.sda.out‚ 856.65234375 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 14483.40625 disk.sdb1.in‚0 disk.sda2.in‚128.625$ memory_percent‚95.98493576049805 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 856.65234375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 network.recv‚ 1.197484e+06 disk.rbd0.in‚0 disk.md0.out‚ 842.67578125 cpu‚25.204989581316976 network.sent‚ 6.390549e+06 disk.nvme0n1.out‚ 39832.0625! proc.memory.rssMB‚ 2986.390625ÉØ¢ä’ú‚(:‡ ¤‹ëÔöΔá gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡·Ç>û‚(:³ ³‹ëÔ«˜£à disk.sda.out‚858.50830078125 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.md0.out‚ 844.4765625 disk.nvme1n1.in‚0! proc.memory.rssMB‚ 2986.390625 cpu‚25.204177878516166 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚95.98350524902344$ proc.memory.availableMB‚658.0625 network.sent‚ 6.49959e+06 network.recv‚ 1.203882e+06! disk.nvme0n1.out‚ 40046.515625! disk.sda2.out‚858.50830078125 disk.dm-0.in‚0 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚858.50830078125 disk.md0p1.out‚ 844.4765625$ disk./.usageGB‚2715.407371520996 disk.nvme0n1.in‚0 disk.sda2.in‚128.625" disk.nvme1n1.out‚ 14483.5703125 disk.sdb1.in‚0) disk./.usagePercent‚75.92392659400325 disk.rbd0.out‚1.125* proc.memory.percent‚18.227481842041016 disk.dm-0.out‚ 54530.0859375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda1.out‚0 disk.sdb.out‚858.50830078125\ò’“ü‚(:ˆ ³‹ëÔ¬”úà gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ór“ý‚(:ˆ ‹ëÔàÓõÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉJÓ>þ‚(:³ ‹ëÔÙ÷Ëà network.recv‚ 1.207927e+06 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 845.85546875 network.sent‚ 6.505249e+06$ disk./.usageGB‚2715.531982421875# disk.nvme1n1.out‚14483.84765625 disk.md0p1.out‚ 845.85546875 disk.sdb1.out‚0 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0$ proc.memory.availableMB‚657.8125) disk./.usagePercent‚75.92741076694554 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2986.390625 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.sda.out‚859.9443359375 disk.rbd0.out‚1.125 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sdb2.out‚859.9443359375$ memory_percent‚95.98503112792969 disk.rbd2.out‚1.0625 disk.nvme0n1.out‚ 40286.09375 disk.sda2.out‚859.9443359375 disk.dm-0.out‚54769.94140625 disk.sdb.out‚859.9443359375* proc.memory.percent‚18.227481842041016 cpu‚25.203363457064647nÐdi’ÿ‚(:‡ Ñ‹ëÔá‚ÊÞ gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„BéuE€‚(:º Ñ‹ëÔãÊžà network.sent‚ 6.511455e+06 disk.nvme1n1.in‚0 disk.md0p1.out‚ 848.50390625 disk.sdb1.in‚0& proc.memory.availableMB‚ 657.796875 disk.sda1.out‚0 disk.md0.out‚ 848.50390625 disk.sda.in‚128.625 disk.sda.out‚862.63916015625 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.dm-0.out‚55021.54296875 disk.sdb.out‚862.63916015625 proc.cpu.threads‚17 network.recv‚ 1.212618e+06) disk./.usagePercent‚75.93442934994074# disk.nvme1n1.out‚14483.97265625 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚95.98512649536133! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0" disk.nvme0n1.out‚ 40537.5703125 disk.sda1.in‚0 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚862.63916015625! disk.sda2.out‚862.63916015625 cpu‚25.20263773970426$ disk./.usageGB‚2715.783000946045Þ„(2- _wandbruntime‚3467  _runtime‚3467ÿĘÄ‚¾ % lr-AdamW/pg1‚0.000689098890170819 % lr-AdamW/pg2‚0.000689098890170819 trainer/global_step‚2899 $ _timestamp‚1.788528083536125e+09 _runtime‚3467.071605905 _step‚234r>ÇɃà % lr-AdamW/pg1‚0.000689098890170819 % lr-AdamW/pg2‚0.000689098890170819 trainer/global_step‚2899 $ _timestamp‚1.788528083536125e+09 _runtime‚3467.071605905 _step‚234êBƒÙ2„- _wandbruntime‚3467  _runtime‚3467¶s‰’ñ'…ëO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3315368915968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3314458945024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235526565085968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73308245 _step‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36200360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60424768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268765245114880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3465546131968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243556079974912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272230791246848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580067 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272231050260992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60424768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3302062471203 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246871448890880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246870520916992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3315368915968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3302970446203 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238828610517811 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53781064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 % _timestamp‚1.7885280835370977e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73334588 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243556061971968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268765269054976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0(Ö´° †ðO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60424768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3302062471203 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246871448890880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246870520916992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3315368915968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.alš™èFlocated‚ 3302970446203 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238828610517811 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53781064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 % _timestamp‚1.7885280835370977e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73334588 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243556061971968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268765269054976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3315368915968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3314458945024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235526565085968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73308245 _step‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36200360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60424768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268765245114880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3465546131968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243556079974912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272230791246848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580067 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272231050260992ë†:2‡- _wandbruntime‚3468  _runtime‚3468`ªËLû&ˆõM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73359445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36212427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268855470819328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36212425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73359445 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚2899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60444928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _runtime‚3468.173342625 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243637681057792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73333076 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3466723270144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3303179744481 $ _timestamp‚1.788528084637262e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243637699060736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60444930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243637699060736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272322194089472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235605565641152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246954189450240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3303179744481 $ _timestamp‚1.788528084637262e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243637699060736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60444930 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243637699060736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272322194089472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235605565641152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246954189450240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268855470819328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36212425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73359445 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚2899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60444928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _runtime‚3468.173342625 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243637681057792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73333076 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3466723270144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ìóv2Š- _wandbruntime‚3468  _runtime‚3468Öw²ÔU‹Ï 2loss/SRqqt3/assignment_loss‚0.8807827831421158 1loss/FBt2/assignment_loss‚0.12582156778755824 /loss/FRt1/detection_loss‚3.0514176988601687 /loss/FRt3/detection_loss‚1.3921825368371274 1loss/SRqqt2/detection_loss‚2.8283594020388345 % _timestamp‚1.7885280846397545e+09 1loss/SRqqt2/assignment_loss‚1.835278790105473 1loss/FBt4/assignment_loss‚0.12129184655145764 0loss/SRqqt1/detection_loss‚2.414211434346658 %loss/total_loss‚44.21559072820042 2loss/SRqqt4/assignment_loss‚1.6150317034300636 .loss/FRt2/detection_loss‚3.603380411863327 /loss/FBt3/detection_loss‚0.6347539109033598 epoch‚2 /loss/FRt3/assignment_loss‚1.155928725997607 trainer/global_step‚2899 /loss/FBt1/assignment_loss‚0.221637312879011 .loss/FBt2/detection_loss‚2.019660961978576 1loss/SRqqt1/assignment_loss‚2.209817716092975 2loss/FBt3/assignment_loss‚0.042656037910453255 .loss/FBt1/detection_loss‚2.225230887532234 _runtime‚3468.175074893 .loss/FRt4/assignment_loss‚3.41537709946328 /loss/FRt4/detection_loss‚2.3491167451472994 1loss/SRqqt3/detection_loss‚0.9686809973699031 /loss/FRt1/assignment_loss‚3.197850387096405 0loss/SRqqt4/detection_loss‚2.109387205365826 .loss/FBt4/detection_loss‚2.117286901921034 _step‚237 /loss/FRt2/assignment_loss‚3.680447663579668HRÍZŒÔ epoch‚2 /loss/FRt3/assignment_loss‚1.155928725997607 trainer/global_step‚2899 /loss/FBt1/assignment_loss‚0.221637312879011 .loss/FBt2/detection_loss‚2.019660961978576 1loss/SRqqt1/assignment_loss‚2.209817716092975 2loss/FBt3/assignment_loss‚0.042656037910453255 .loss/FBt1/detection_loss‚2.225230887532234 _runtime‚3468.175074893 .loss/FRt4/assignment_loss‚3.41537709946328 /loss/FRt4/detection_loss‚2.3491167451472994 1loss/SRqqt3/detection_loss‚0.9686809973699031 /loss/FRt1/assignment_loss‚3.197850387096405 0loss/SRqqt4/detection_loss‚2.109387205365826 .loss/FBt4/detection_loss‚2.117286901921034 _step‚237 /loss/FRt2/assignment_loss‚3.680447663579668 2loss/SRqqt3/assignment_loss‚0.8807827831421158 1loss/FBt2/assignment_loss‚0.12582156778755824 /loss/FRt1/detection_loss‚3.0514176988601687 /loss/FRt3/detection_loss‚1.3921825368371274 1loss/SRqqt2/detection_loss‚2.8283594020388345 % _timestamp‚1.7885280846397545e+09 1loss/SRqqt2/assignment_loss‚1.835278790105473 1loss/FBt4/assignment_loss‚0.12129184655145764 0loss/SRqqt1/detection_loss‚2.414211434346658 %loss/total_loss‚44.21559072820042 2loss/SRqqt4/assignment_loss‚1.6150317034300636 .loss/FRt2/detection_loss‚3.603380411863327 /loss/FBt3/detection_loss‚0.6347539109033598íèÂñ“‚(:ˆ à‹ëÔÉì¿à gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B2¿.AŽ‚(:¶ à‹ëÔÔÑ‘â! disk.sda2.out‚864.32568359375 disk.sdb.in‚ 70.9609375$ memory_percent‚95.98531723022461& proc.memory.availableMB‚ 657.765625 disk.rbd0.out‚1.125 disk.nvme0n1.in‚0# disk.nvme0n1.out‚40776.10546875 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.sdb.out‚864.32568359375! disk.sdb2.out‚864.32568359375 disk.sda.in‚128.625 disk.md0p1.out‚ 850.140625 network.sent‚ 6.527396e+06" disk.nvme1n1.out‚ 14484.2421875 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.20181295888878 disk.sda2.in‚128.625 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚1.0625 disk.md0.out‚ 850.140625 disk.dm-0.in‚0 disk.dm-0.out‚55260.34765625 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 network.recv‚ 1.216221e+06) disk./.usagePercent‚75.93440247148665$ disk./.usageGB‚2715.782039642334 disk.sda.out‚864.32568359375! proc.memory.rssMB‚ 2986.390625 disk.sdb1.in‚0 disk.sdb1.out‚0µÃÈ;”‚(:‰ ï‹ëÔÿí¸à gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.099' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.27475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tKl[÷‚(:¯ ï‹ëÔëïÊá disk.md0p1.out‚851.5( proc.memory.availableMB‚ 658.01953125 disk.sdb.in‚ 70.9609375! disk.sda2.out‚865.74755859375 disk.md0.in‚ 199.5859375 disk.md0.out‚851.5 disk.dm-0.in‚0" disk.nvme0n1.obüšÌCut‚ 41166.5078125! disk.sdb2.out‚865.74755859375 disk.sda.in‚128.625 disk.sda.out‚865.74755859375 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚1.125 disk.rbd2.out‚1.0625 cpu‚25.201103185034974 proc.cpu.threads‚17 network.sent‚ 6.621365e+06 disk.sda2.in‚128.625 disk.sdb1.in‚0 disk.rbd2.in‚0 network.recv‚ 1.222649e+06 disk.sdb.out‚865.74755859375# disk.nvme1n1.out‚14484.43359375 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚75.91965739954455$ disk./.usageGB‚2715.254684448242 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚55650.94140625 disk.sdb1.out‚0$ memory_percent‚95.98376750946045! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 disk.sdb2.in‚ 70.9609375žçøõž‘‚(:“ þ‹ëÔ¾¦¤à gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[mªŽA’‚(:¶ þ‹ëÔ¦ç‹á disk.sda.out‚867.45556640625 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.sda2.in‚128.625 disk.sdb.out‚867.45556640625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚867.45556640625 cpu‚25.200398421190837 disk.sdb.in‚ 70.9609375) disk./.usagePercent‚75.91841459102426 disk.nvme1n1.in‚0 disk.md0p1.out‚ 853.140625 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 853.140625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2986.390625 network.recv‚ 1.226631e+06$ disk./.usageGB‚2715.210235595703 disk.sda.in‚128.625* proc.memory.percent‚18.227481842041016 disk.rbd0.out‚1.125 disk.dm-0.out‚55852.46484375$ memory_percent‚95.98333835601807 network.sent‚ 6.629082e+06# disk.nvme1n1.out‚14484.65234375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.out‚ 41367.8125! disk.sda2.out‚867.45556640625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.dm-0.in‚0( proc.memory.availableMB‚ 658.08984375üÓ‰ 2“- _wandbruntime‚3524  _runtime‚35245ûµ›Ç”Á & lr-AdamW/pg2‚0.0006865889677373121 trainer/global_step‚2949 % _timestamp‚1.7885281407357337e+09 _runtime‚3524.271227669 _step‚238 & lr-AdamW/pg1‚0.0006865889677373121SY÷ÎÌ•Æ % _timestamp‚1.7885281407357337e+09 _runtime‚3524.271227669 _step‚238 & lr-AdamW/pg1‚0.0006865889677373121 & lr-AdamW/pg2‚0.0006865889677373121 trainer/global_step‚2949î`J"Ò2–- _wandbruntime‚3524  _runtime‚3524Å\ÐUñ'—ëO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3371456340992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3357939726311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54673760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242835367500119 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚239 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61432770 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239477427773808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚3524.272330554 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74553147 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚242836292514479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3370546370048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74579490 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216tù¾fö'˜ðO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54673760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242835367500119 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚239 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61432770 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239477427773808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚3524.272330554 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74553147 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚242836292514479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3370546370048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74579490 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3371456340992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3357939726311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ’™‚(:‡ ŒëÔÂìÂß gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§—tœHš‚(:½ ŒëÔúÕüß) disk./.usagePercent‚75.92031314849604 disk.sda1.out‚0 disk.sdb.out‚868.53662109375 disk.rbd2.out‚1.0625 disk.md0.in‚ 199.5859375 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.out‚ 854.16796875 disk.dm-0.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚14556.71484375 disk.md0p1.out‚ 854.16796875% disk./.usageGB‚2715.2781372070312 disk.sda2.in‚128.625! disk.sdb2.out‚868.53662109375 proc.cpu.threads‚17 network.recv‚ 1.231601e+06" disk.nvme0n1.out‚ 41523.8203125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 disk.rbd0.in‚0 disk.rbd0.out‚1.125! disk.sda2.out‚868.53662109375 disk.dm-0.out‚56080.53515625 disk.sda.in‚128.625 disk.sda.out‚868.53662109375' proc.memory.availableMB‚ 657.8359375! proc.memory.rssMB‚ 2986.390625 network.sent‚ 6.633128e+06$ memory_percent‚95.98488807678223 cpu‚25.1996385374609632µ¾2›- _wandbruntime‚3525  _runtime‚35251ŒÔË™œöM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239556462407712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17876020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247718653351936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276892255108608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74604347 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239556445368352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61452930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 % _timestamp‚1.7885281417358615e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚3525.273261272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242916427382509 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_t â;crain_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136030906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3372577814528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247718635348992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74577978 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273366779981824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚2949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36815675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273366756041728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928NƒØX'ûM trainer/global_step‚2949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36815675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273366756041728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239556462407712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17876020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247718653351936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276892255108608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74604347 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239556445368352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61452930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 % _timestamp‚1.7885281417358615e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚3525.273261272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242916427382509 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136030906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3372577814528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247718635348992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74577978 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273366779981824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712ð®,¥a2ž- _wandbruntime‚3525  _runtime‚3525k¸)lZŸÔ /loss/FRt4/detection_loss‚2.3960869193077086 _step‚241 %loss/total_loss‚46.78979240829406 .loss/FBt4/detection_loss‚2.121487871596688 0loss/FRt3/assignment_loss‚1.8031402004176174 0loss/FRt4/assignment_loss‚3.6428304955363275 1loss/SRqqt4/detection_loss‚2.0792184697615133 0loss/SRqqt2/detection_loss‚2.558793925813266 0loss/FRt1/assignment_loss‚2.8540063296045575 1loss/SRqqt1/assignment_loss‚2.195335362966244 1loss/FBt1/assignment_loss‚0.21274385928061107 1loss/FBt2/assignment_loss‚0.23262076556089928 0loss/FBt3/assignment_loss‚0.2017057493343528 2loss/SRqqt4/assignment_loss‚1.6014274781620181 _runtime‚3525.274772769 /loss/FRt1/detection_loss‚2.7870755076408384 .loss/FBt1/detection_loss‚1.513745353834049 /loss/FBt3/detection_loss‚0.7406582663188109 trainer/global_step‚2949 2loss/SRqqt3/assignment_loss‚1.1823246396940064 1loss/FBt4/assignment_loss‚0.40402045340052656 .loss/FRt2/detection_loss‚2.749879793116921 /loss/FRt3/detection_loss‚1.6910338663849338 1loss/SRqqt3/detection_loss‚1.1696169094217044 1loss/SRqqt1/detection_loss‚2.5143979581502767 .loss/FBt2/detection_loss‚3.439342431046746 epoch‚2 % _timestamp‚1.7885281417392328e+09 .loss/FRt2/assignment_loss‚4.47278391373785 2loss/SRqqt2/assignment_loss‚2.2255158882055963ä£c_ Ù 2loss/SRqqt3/assignment_loss‚1.1823246396940064 1loss/FBt4/assignment_loss‚0.40402045340052656 .loss/FRt2/detection_loss‚2.749879793116921 /loss/FRt3/detection_loss‚1.6910338663849338 1loss/SRqqt3/detection_loss‚1.1696169094217044 1loss/SRqqt1/detection_loss‚2.5143979581502767 .loss/FBt2/detection_loss‚3.439342431046746 epoch‚2 % _timestamp‚1.7885281417392328e+09 .loss/FRt2/assignment_loss‚4.47278391373785 2loss/SRqqt2/assignment_loss‚2.2255158882055963 /loss/FRt4/detection_loss‚2.3960869193077086 _step‚241 %loss/total_loss‚46.78979240829406 .loss/FBt4/detection_loss‚2.121487871596688 0loss/FRt3/assignment_loss‚1.8031402004176174 0loss/FRt4/assignment_loss‚3.6428304955363275 1loss/SRqqt4/detection_loss‚2.0792184697615133 0loss/SRqqt2/detection_loss‚2.558793925813266 0loss/FRt1/assignment_loss‚2.8540063296045575 1loss/SRqqt1/assignment_loss‚2.195335362966244 1loss/FBt1/assignment_loss‚0.21274385928061107 1loss/FBt2/assignment_loss‚0.23262076556089928 0loss/FBt3/assignment_loss‚0.2017057493343528 2loss/SRqqt4/assignment_loss‚1.6014274781620181 _runtime‚3525.274772769 /loss/FRt1/detection_loss‚2.7870755076408384 .loss/FBt1/detection_loss‚1.513745353834049 /loss/FBt3/detection_loss‚0.7406582663188109 trainer/global_step‚2949ñz^53”¡‚(:‰ œŒëÔÞ¤£á gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%1‰0>¢‚(:³ œŒëÔ§¡åá disk.sda2.in‚128.625 network.sent‚ 6.74253e+06 network.recv‚ 1.238073e+06 disk.sdb.in‚ 70.9609375 disk.rbd2.out‚1.0625* proc.memory.percent‚18.227481842041016 cpu‚25.198911587422955 disk.md0p1.in‚ 199.5859375% disk./.usageGB‚2715.3435287475586 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 56126.1953125! proc.memory.rssMB‚ 2986.390625 disk.sdb.out‚870.22998046875! disk.sdb2.out‚870.22998046875 disk.sda.in‚128.625 disk.rbd2.in‚0" disk.nvme0n1.out‚ 41548.0703125 disk.md0.out‚ 855.8046875' proc.memory.availableMB‚ 657.5546875 disk.sdb1.in‚0! disk.sda2.out‚870.22998046875 disk.nvme1n1.out‚14585.25 disk.md0p1.out‚ 855.8046875 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.in‚0$ memory_percent‚95.98660469055176) disk./.usagePercent‚75.92214152333766 disk.sda.out‚870.22998046875 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.nvme0n1.in‚0n.#?£‚(:´ «ŒëÔ„¬à! disk.nvme1n1.out‚ 14637.078125 disk.rbd0.in‚0 disk.sdb1.out‚0! proc.memory.rssMB‚ 2986.390625 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.sdb.in‚ 70.9609375 disk.sda.out‚871.59619140625# memory_percent‚95.9850549697876 disk.sda2.in‚128.625 disk.md0.out‚ 857.1171875 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 857.1171875 disk.sdb1.in‚0 network.recv‚ 1.24431e+06 disk.sda1.in‚0! disk.sda2.out‚871.59619140625 disk.md0.in‚ 199.5859375 disk.sdb.out‚871.59619140625 proc.cpu.threads‚17) disk./.usagePercent‚75.92739455454466 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚1.125( proc.memory.availableMB‚ 657.80859375 cpu‚25.198243243017835 network.sent‚ 6.828403e+06% disk./.usageGB‚2715.5314025878906 disk.dm-0.out‚ 56372.046875* proc.memory.percent‚18.227481842041016 disk.nvme0n1.out‚ 41734.96875 disk.dm-0.in‚0 disk.sda1.out‚0! disk.sdb2.out‚871.59619140625 disk.sda.in‚128.625ãñvÔ”¤‚(:‰ «ŒëÔ¬ì±ù gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ýo“¥‚(:ˆ ºŒëÔñçà gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i9\¡5¦‚(:ª ºŒëÔ¯ûœá disk.sdb.in‚ 70.9609375) disk./.usagePercent‚75.93439937833122 disk.md0.in‚ 199.5859375 disk.sdb2.out‚ 873.05859375$ memory_percent‚95.98655700683594! proc.memory.rssMB‚ 2986.390625 disk.rbd2.in‚0 network.sent‚ 6.834003e+06 network.recv‚ 1.248218e+06 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sda1.in‚0 disk.sdb.out‚ 873.05859375 disk.sdb1.out‚0# disk.nvme0n1.out‚41892.41796875 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sda2.out‚ 873.05859375 disk.dm-0.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚18.227481842041016# disk.nvme1n1.out‚14766.48828125 disk.rbd2.out‚1.0625% disk./.usageGB‚2715.7819290161133 disk.sda1.out‚0 disk.md0.out‚ 858.5234375 disk.dm-0.out‚ 56658.90625 disk.md0p1.out‚ 858.5234375$ proc.memory.availableMB‚657.5625 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 873.05859375 disk.nvme1n1.in‚0 cpu‚25.197537685299633¤ÃòO2§- _wandbruntime‚3581  _runtime‚35814ÃʙƨÀ & lr-AdamW/pg1‚0.0006840122141488771 & lr-AdamW/pg2‚0.0006840122141488771 trainer/global_step‚2999 % _timestamp‚1.7885281978124197e+09 _runtime‚ 3581.34873417 _step‚242âïâË©Å & lr-AdamW/pg2‚0.0006840122141488771 trainer/global_step‚2999 % _timestamp‚1.7885281978124197e+09 _runtime‚ 3581.34873417 _step‚242 & lr-AdamW/pg1‚0.0006840122141488771ò&7¾12ª- _wandbruntime‚3581  _runtime‚3581?)Ññ'«ëO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251717988557312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251717970554368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62440768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75824392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582911078912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3414724956419 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246842124482427 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3413816981419 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55567436 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277787839499776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243428307501008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138265162 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 $ _timestamp‚1.788528197813233e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62440768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37406860 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243428324540368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255144604349440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75798049 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37406862 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255145532323328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3581.349243202 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18160574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615mÛ™6ç ¬ðO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55567436 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277787839499776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243428307501008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138265162 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 $ _timestamp‚1.788528197813233e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62440768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37406860 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243428324540368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255144604349440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75798049 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/all† G-ocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37406862 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255145532323328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3581.349243202 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18160574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251717988557312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251717970554368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62440768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75824392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582911078912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3414724956419 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246842124482427 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3413816981419ó­êt2­- _wandbruntime‚3582  _runtime‚3582C#i£ü&®öM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75849249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18166340 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138283808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138310179 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3583839757312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885281989654438e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75822880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251799589640192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62460928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚2999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55585267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414934254697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138283808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75822880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277878065204224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251799589640192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3428665239552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246923184364817 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243507342134912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚3582.503851687 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251799607643136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277878041264128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256Õy~‚'¯ûM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚2999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55585267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414934254697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138283808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75822880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277878065204224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251799589640192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3428665239552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246923184364817 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243507342134912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚3582.503851687 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251799607643136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277878041264128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75849249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18166340 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138283808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138310179 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3583839757312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885281989654438e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75822880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251799589640192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62460928ôR‡û2°- _wandbruntime‚3582  _runtime‚3582&ˆ„œV±Ð 0loss/SRqqt2/assignment_loss‚1.80613460438326 1loss/FBt1/assignment_loss‚0.09189895464915025 1loss/FBt4/assignment_loss‚0.22564789004371733 %loss/total_loss‚45.94595829254384 _step‚245 /loss/FRt2/assignment_loss‚4.146628673429842 3loss/FBt3/assignment_loss‚0.0008770719821337991 /loss/FBt3/detection_loss‚0.7954204580187798 % _timestamp‚1.7885281989699337e+09 1loss/SRqqt2/detection_loss‚1.8787657599896193 epoch‚2 ,loss/FRt2/detection_loss‚2.2902477409432 /loss/FRt1/assignment_loss‚3.727916303802939 .loss/FRt4/detection_loss‚2.718366434940925 1loss/SRqqt1/detection_loss‚2.7362794302977047 .loss/FBt2/detection_loss‚2.399589369171544 0loss/FRt3/assignment_loss‚1.4122049612243002 2loss/SRqqt1/assignment_loss‚1.5928124538980997 0loss/FBt2/assignment_loss‚0.2864857585265877 1loss/SRqqt3/detection_loss‚1.0193453292955053 1loss/SRqqt4/detection_loss‚2.5152125358581543 .loss/FRt1/detection_loss‚2.884409233051188 /loss/FRt4/assignment_loss‚4.115243569398538 2loss/SRqqt3/assignment_loss‚1.0720705285964702 .loss/FRt3/detection_loss‚1.347757272967478 1loss/SRqqt4/assignment_loss‚2.094800390941756 .loss/FBt1/detection_loss‚2.484242918093999 /loss/FBt4/detection_loss‚2.3036006490389505 trainer/global_step‚2999 _runtime‚3582.505308143ˆà“ê[²Õ 0loss/FBt2/assignment_loss‚0.2864857585265877 1loss/SRqqt3/detection_loss‚1.0193453292955053 1loss/SRqqt4/detection_loss‚2.5152125358581543 .loss/FRt1/detection_loss‚2.884409233051188 /loss/FRt4/assignment_loss‚4.115243569398538 2loss/SRqqt3/assignment_loss‚1.0720705285964702 .loss/FRt3/detection_loss‚1.347757272967478 1loss/SRqqt4/assignment_loss‚2.094800390941756 .loss/FBt1/detection_loss‚2.484242918093999 /loss/FBt4/detection_loss‚2.3036006490389505 trainer/global_step‚2999 _runtime‚3582.505308143 0loss/SRqqt2/assignment_loss‚1.80613460438326 1loss/FBt1/assignment_loss‚0.09189895464915025 1loss/FBt4/assignment_loss‚0.22564789004371733 %loss/total_loss‚45.94595829254384 _step‚245 /loss/FRt2/assignment_loss‚4.146628673429842 3loss/FBt3/assignment_loss‚0.0008770719821337991 /loss/FBt3/detection_loss‚0.7954204580187798 % _timestamp‚1.7885281989699337e+09 1loss/SRqqt2/detection_loss‚1.8787657599896193 epoch‚2 ,loss/FRt2/detection_loss‚2.2902477409432 /loss/FRt1/assignment_loss‚3.727916303802939 .loss/FRt4/detection_loss‚2.718366434940925 1loss/SRqqt1/detection_loss‚2.7362794302977047 .loss/FBt2/detection_loss‚2.399589369171544 0loss/FRt3/assignment_loss‚1.4122049612243002 2loss/SRqqt1/assignment_loss‚1.5928124538980997õ~…~”³‚(:‰ ÉŒëÔë––á gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®õeÉG´‚(:¼ ÉŒëÔˉ±á! disk.sda2.out‚874.91064453125 disk.md0.in‚ 199.5859375 disk.dm-0.out‚56998.58203125) disk./.usagePercent‚75.93437047332702$ memory_percent‚95.98493576049805* proc.memory.percent‚18.227481842041016% disk./.usageGB‚2715.7808952331543 disk.nvme1n1.in‚0 disk.rbd0.out‚1.125 disk.rbd2.in‚0 cpu‚25.196714440566584 network.recv‚ 1.260402e+06 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375& proc.memory.availableMB‚ 657.828125 proc.cpu.threads‚17 disk.sda.out‚874.91064453125 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.sdb.out‚874.91064453125 disk.sda.in‚128.625 disk.sda1.in‚0" disk.nvme0n1.out‚ 42154.4765625 disk.md0.out‚ 860.30859375! proc.memory.rssMB‚ 2986.390625 network.sent‚ 6.855114e+06 disk.md0p1.out‚ 860.30859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚1.0625 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚874.91064453125# disk.nvme1n1.out‚14844.10546875 disk.sda2.in‚128.625´ï{§’µ‚(:‡ ØŒëÔû×ûÞ gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó¯#%¶‚(:³ ØŒëԵőâ disk.sdb2.in‚ 70.9609375" disk.nvme0n1.out‚ 42286.7890625* proc.memory.percent‚18.227481842041016 disk.dm-0.out‚ 57402.5625 disk.sdb.out‚876.3173828125 disk.md0p1.out‚ 861.66796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 YŠb disk.sda2.out‚876.3173828125' proc.memory.availableMB‚ 657.8359375$ disk./.usageGB‚2715.378646850586 disk.sda.out‚876.3173828125 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 861.66796875! proc.memory.rssMB‚ 2986.390625) disk./.usagePercent‚75.92312344019638 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚876.3173828125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 15115.7734375 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 cpu‚25.195958743360364 network.sent‚ 6.952068e+06 network.recv‚ 1.267249e+06 disk.rbd0.out‚1.125 disk.dm-0.in‚0 disk.sda.in‚128.625 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚0 proc.cpu.threads‚17$ memory_percent‚95.98488807678223 âd’·‚(:‡ çŒëÔ¸¯óÞ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.684' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.921 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÅp„C¸‚(:¸ çŒëÔ½ªÁá" disk.nvme0n1.out‚ 42469.2109375 disk.sda.out‚ 877.67578125 disk.sda2.in‚128.625( proc.memory.availableMB‚ 657.84765625 cpu‚25.19516005162256 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.sdb.out‚ 877.67578125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda2.out‚ 877.67578125 network.recv‚ 1.271169e+06 disk.sdb2.out‚ 877.67578125 disk.md0p1.out‚ 862.95703125 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 862.95703125$ disk./.usageGB‚2715.208786010742 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.rbd2.out‚ 1.13671875 disk.md0.in‚ 199.5859375 network.sent‚ 6.955254e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 disk.dm-0.out‚57629.23828125# disk.nvme1n1.out‚15160.02734375 disk.rbd0.out‚ 1.13671875# memory_percent‚95.9848165512085) disk./.usagePercent‚75.91837406002206 disk.dm-0.in‚0øÉü㔹‚(:‰ öŒëÔ«Ì”à gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ǭ`ŸHº‚(:½ öŒëÔô…Ëà disk.md0.in‚ 199.5859375 disk.rbd2.in‚0 cpu‚25.194508510792183 disk.md0p1.out‚ 864.52734375 disk.sdb1.out‚0 disk.rbd2.out‚ 1.13671875 disk.sda1.out‚0 disk.md0.out‚ 864.52734375# disk.nvme1n1.out‚15160.48046875 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sdb.out‚879.2978515625 disk.nvme1n1.in‚0 disk.rbd0.in‚0# memory_percent‚95.9848403930664* proc.memory.percent‚18.227481842041016$ disk./.usageGB‚2715.281108856201 disk.sda2.out‚879.2978515625 disk.sda.out‚879.2978515625 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.19921875 network.sent‚ 6.961859e+06) disk./.usagePercent‚75.92039623705055 disk.dm-0.out‚ 57813.8359375 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 proc.cpu.threads‚17 network.recv‚ 1.27591e+06# disk.nvme0n1.out‚42653.35546875 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚879.2978515625 disk.sdb1.in‚0% proc.memory.availableMB‚ 657.84375! proc.memory.rssMB‚ 2986.390625 disk.sda2.in‚128.625_Ð!2»- _wandbruntime‚3638  _runtime‚3638»BÀǼÁ & lr-AdamW/pg1‚0.0006813691663970998 & lr-AdamW/pg2‚0.0006813691663970998 trainer/global_step‚3049 % _timestamp‚1.7885282546815665e+09 _runtime‚3638.217200421 _step‚246RãCÌ½Æ trainer/global_step‚3049 % _timestamp‚1.7885282546815665e+09 _runtime‚3638.217200421 _step‚246 & lr-AdamW/pg1‚0.0006813691663970998 & lr-AdamW/pg2‚0.0006813691663970998öÃI´þ2¾- _wandbruntime‚3638  _runtime‚3638•ªûò'¿ìO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3470602211527 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚3049 % _timestamp‚1.7885282546831412e+09 _step‚247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255798942848512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285940185486336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77042951 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3483631191040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77069294 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482721220096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285940423528960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140518064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77042951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255798924845568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3638.219055783 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259281646065664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38010112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38010110 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345è8n÷'ÀñO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63448770 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63448768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3483631191040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56460421 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18449690 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3638.219055783 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259281646065664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38010112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38010110 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3470602211527 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚3049 % _timestamp‚1.7885282546831412e+09 _step‚247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255798942848512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285940185486336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77042951 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3483631191040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77069294 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482721220096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285940423528960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140518064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77042951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255798924845568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18456130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250929941347125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3484752664576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250929016332557 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255880561934336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77094151 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642252935680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140536710 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77067782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18455494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259364386611712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 $ _timestamp‚1.788528255862214e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140563081 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259365314598912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483842680320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63468930 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63468930 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255880543931392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247458204822752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3471719485013 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0=O§Z'ÃúM _step‚248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3484752664576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642481704960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255880561934336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚3639.399006178 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286031832131584 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259365314598912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483842680320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63468930 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63468930 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255880543931392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247458204822752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3471719485013 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3470811509805 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483842680320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63468928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140536710 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63468928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38022177 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77094151 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18456130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250929941347125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3484752664576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250929016332557 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255880561934336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77094151 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642252935680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140536710 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77067782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18455494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259364386611712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 $ _timestamp‚1.788528255862214e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140563081 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286810386004480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539718616064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286810409944576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3695.784142546 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3525571491635 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287853 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259879879136768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259879897139712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944a£ƒ%ðÔñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142770966 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78314196 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229654016 trainer/global_step‚3099 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78287853 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserj£wÌ$ved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64456768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539718616064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251330083994768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259879897139712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286810386004480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539718616064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286810409944576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3695.784142546 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3525571491635 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287853 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259879879136768 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254935773314865 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚3696.782622069 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142789612 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78339053 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64476930 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3701052502528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233835008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540840089600 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286900635649024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259961498222592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263501428327936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290601898046464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0rò`™'×ûM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290601898046464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142815983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259961516225536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78312684 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57371817 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142789612 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251409084549952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259961516225536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254936698329433 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38625425 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64476928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263502356315136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78339053 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254935773314865 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚3696.782622069 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142789612 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78339053 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64476930 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3701052502528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233835008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540840089600 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286900635649024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259961498222592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263501428327936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794üFä}K2Ø- _wandbruntime‚3696  _runtime‚3696Ñð¼UÙÏ trainer/global_step‚3099 1loss/SRqqt2/detection_loss‚2.6453888535499575 1loss/SRqqt3/detection_loss‚1.1459233376609033 0loss/FBt4/assignment_loss‚0.5491257985739659 .loss/FRt4/detection_loss‚2.322323182887501 /loss/FBt1/detection_loss‚2.9082912653684616 $loss/total_loss‚47.7868509673105 _step‚253 1loss/SRqqt1/assignment_loss‚2.108558756609758 2loss/SRqqt2/assignment_loss‚2.3290887373685836 2loss/SRqqt3/assignment_loss‚1.2152185400861293 0loss/SRqqt1/detection_loss‚2.901297894616922 epoch‚2 2loss/SRqqt4/assignment_loss‚1.4651529482432775 1loss/FBt1/assignment_loss‚0.19419589268852633 0loss/FBt2/assignment_loss‚0.1051668346303245 .loss/FRt3/detection_loss‚1.421461324310965 /loss/FBt3/detection_loss‚0.6459631007890423 _runtime‚3696.784095156 /loss/FRt2/assignment_loss‚4.621948258630161 0loss/FRt3/assignment_loss‚1.5217291257447667 1loss/FBt3/assignment_loss‚0.03590234359036792 /loss/FBt2/detection_loss‚2.3440132085233927 .loss/FBt4/detection_loss‚2.728097357920238 /loss/FRt1/assignment_loss‚3.368592998811177 .loss/FRt1/detection_loss‚2.977293223142624 -loss/FRt2/detection_loss‚2.64710350283261 0loss/SRqqt4/detection_loss‚2.137294383559908 0loss/FRt4/assignment_loss‚3.4477200971709356 % _timestamp‚1.7885283132488227e+09À!ZÚÔ 0loss/SRqqt1/detection_loss‚2.901297894616922 epoch‚2 2loss/SRqqt4/assignment_loss‚1.4651529482432775 1loss/FBt1/assignment_loss‚0.19419589268852633 0loss/FBt2/assignment_loss‚0.1051668346303245 .loss/FRt3/detection_loss‚1.421461324310965 /loss/FBt3/detection_loss‚0.6459631007890423 _runtime‚3696.784095156 /loss/FRt2/assignment_loss‚4.621948258630161 0loss/FRt3/assignment_loss‚1.5217291257447667 1loss/FBt3/assignment_loss‚0.03590234359036792 /loss/FBt2/detection_loss‚2.3440132085233927 .loss/FBt4/detection_loss‚2.728097357920238 /loss/FRt1/assignment_loss‚3.368592998811177 .loss/FRt1/detection_loss‚2.977293223142624 -loss/FRt2/detection_loss‚2.64710350283261 0loss/SRqqt4/detection_loss‚2.137294383559908 0loss/FRt4/assignment_loss‚3.4477200971709356 % _timestamp‚1.7885283132488227e+09 trainer/global_step‚3099 1loss/SRqqt2/detection_loss‚2.6453888535499575 1loss/SRqqt3/detection_loss‚1.1459233376609033 0loss/FBt4/assignment_loss‚0.5491257985739659 .loss/FRt4/detection_loss‚2.322323182887501 /loss/FBt1/detection_loss‚2.9082912653684616 $loss/total_loss‚47.7868509673105 _step‚253 1loss/SRqqt1/assignment_loss‚2.108558756609758 2loss/SRqqt2/assignment_loss‚2.3290887373685836 2loss/SRqqt3/assignment_loss‚1.2152185400861293ý±|F”Û‚(:‰ ÁëÔäØà gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýì×PÜ‚(:Ä ÁëÔ¬øá disk.nvme1n1.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625! disk.sda2.out‚886.54541015625# disk.nvme1n1.out‚15161.98828125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.34130859375 disk.rbd2.out‚ 1.341796875 disk.dm-0.out‚ 58879 º9.53125 disk.md0.out‚ 871.4765625 disk.md0p1.out‚ 871.4765625 network.recv‚ 1.326772e+06) disk./.usagePercent‚75.93437292651926 disk.sdb.out‚886.54541015625 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.227481842041016% disk./.usageGB‚2715.7809829711914 disk.sda.out‚886.54541015625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0# disk.nvme0n1.out‚43717.54296875 network.sent‚ 7.12045e+06 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.sdb1.in‚0$ memory_percent‚95.98569869995117& proc.memory.availableMB‚ 657.703125 proc.cpu.threads‚17! disk.sdb2.out‚886.54541015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2986.390625 cpu‚25.191180640948204O9¼”Ý‚(:‰ ÐëÔŸê­à gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&DgÀKÞ‚(:À ÐëÔŠìáà network.recv‚ 1.338818e+06 disk.nvme0n1.out‚ 43970.9375 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda.out‚887.72998046875$ memory_percent‚95.98710536956787! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 network.sent‚ 7.21924e+06 disk.nvme0n1.in‚0 disk.sdb.out‚887.72998046875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 872.59765625( proc.memory.availableMB‚ 657.47265625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.in‚0! disk.sda2.out‚887.72998046875 disk.nvme1n1.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚887.72998046875 disk.sda.in‚128.625) disk./.usagePercent‚75.91962476142172 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.in‚0! disk.nvme1n1.out‚ 15162.453125$ disk./.usageGB‚2715.253517150879 disk.sda2.in‚128.625 disk.sda1.in‚0 disk.dm-0.out‚ 59133.390625 disk.sdb2.in‚ 70.9609375 cpu‚25.190564829984943 disk.rbd2.out‚ 1.341796875 disk.md0.out‚ 872.59765625a(±v”ß‚(:‰ ßëÔØ–Ðà gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚150.391' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.59775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òpC¶Qà‚(:Æ ßëÔÛ‡ºá* proc.memory.percent‚18.227481842041016 network.sent‚ 7.223832e+06) disk./.usagePercent‚75.91837427334312 disk.rbd0.out‚ 1.404296875 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 disk.sda1.in‚0! disk.sda2.out‚889.34228515625( proc.memory.availableMB‚ 657.97265625 proc.cpu.threads‚17% disk./.usageGB‚2715.2087936401367 disk.sdb1.out‚0 disk.md0.out‚ 874.14453125 disk.dm-0.in‚0! disk.sdb2.out‚889.34228515625# disk.nvme1n1.out‚15162.69140625 disk.rbd0.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚44115.44921875 disk.sda2.in‚128.625 disk.sda.out‚889.34228515625! proc.memory.rssMB‚ 2986.390625 network.recv‚ 1.34251e+06 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0 disk.dm-0.out‚ 59278.140625 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 cpu‚25.18992776809162 disk.sda1.out‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚ 874.14453125$ memory_percent‚95.98405361175537 disk.rbd2.out‚ 1.341796875 disk.sdb.out‚889.34228515625 disk.sda.in‚128.625×8’á‚(:‡ îëÔªµóà gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.4825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H’RNâ‚(:à îëÔ›–§á disk.sdb.in‚ 70.9609375! disk.sdb2.out‚890.66845703125 disk.sda.in‚128.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.341796875! disk.sda2.out‚890.66845703125 disk.sda.out‚890.66845703125# disk.nvme1n1.out‚15234.83984375 disk.md0.in‚ 199.5859375! proc.memory.rssMB‚ 2986.390625* proc.memory.percent‚18.227481842041016 network.sent‚ 7.231513e+06 network.recv‚ 1.346251e+06 disk.nvme0n1.in‚0$ disk./.usageGB‚2715.281093597412 disk.nvme1n1.in‚0% proc.memory.availableMB‚ 657.96875 proc.cpu.threads‚17 disk.rbd0.out‚ 1.404296875$ memory_percent‚95.98407745361328 disk.rbd2.in‚0 disk.sda2.in‚128.625) disk./.usagePercent‚75.92039581040842 disk.dm-0.in‚0 disk.dm-0.out‚ 59468.484375 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 875.41015625 disk.sdb1.in‚0 disk.sdb.out‚890.66845703125 disk.rbd0.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚44233.64453125 cpu‚25.18930734272354 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 875.41015625Lô2ã- _wandbruntime‚3753  _runtime‚3753îp4bÆäÀ _step‚254 & lr-AdamW/pg1‚0.0006758864053330945 & lr-AdamW/pg2‚0.0006758864053330945 trainer/global_step‚3149 $ _timestamp‚1.788528369628979e+09 _runtime‚3753.164509219÷ù#IËåÅ trainer/global_step‚3149 $ _timestamp‚1.788528369628979e+09 _runtime‚3753.164509219 _step‚254 & lr-AdamW/pg1‚0.0006758864053330945 & lr-AdamW/pg2‚0.0006758864053330945þÐm­Û2æ- _wandbruntime‚3753  _runtime‚3753^}W^ò'çìO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216612 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267556657472000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267556657472000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295080592207360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 trainer/global_step‚3149 % _timestamp‚1.7885283696296976e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3595806041088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3594896070144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65464770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3758897040384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263960851430912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637iÌ–÷'èñO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65464770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3758897040384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263960851430912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216612 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267556657472000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267556657472000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295080592207360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 trainer/global_step‚3149 % _timestamp‚1.7885283696296976e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3595806041088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3594896070144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784ÿò2é- _wandbruntime‚3754  _runtime‚3754 ý°\ªêöM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79557586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759826865664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79557586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267639398031360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255359964277152 % _timestamp‚1.7885283705833223e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19035552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3583473995229 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582566020021 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 trainer/global_step‚3149 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3596017530368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 VIDeviceStatsMonitor.o”.iRn_train_batch_end/inactive_split.large_pool.allocated‚39228677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3596927514624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65484928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79583955 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570m7'ëûM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79583955 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79557586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759826865664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79557586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267639398031360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255359964277152 % _timestamp‚1.7885283705833223e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19035552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3583473995229 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582566020021 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 trainer/global_step‚3149 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3596017530368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39228677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3596927514624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65484928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0€™k«À2ì- _wandbruntime‚3754  _runtime‚3754ö‚(:³ ªŽëÔºæ“á disk.nvme1n1.out‚ 15599.40625 network.sent‚ 7.366331e+06 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚ 896.7578125 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 network.recv‚ 1.383061e+06* proc.memory.percent‚18.227481842041016 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb2.out‚ 896.7578125 disk.sda.in‚128.625 cpu‚25.186868582380946 disk.sda1.out‚0 disk.dm-0.out‚60462.44140625 disk.md0p1.in‚ 199.5859375$ disk./.usageGB‚2715.780975341797 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚44863.03515625 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 657.515625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sdb.out‚ 896.7578125$ memory_percent‚95.98684310913086! proc.memory.rssMB‚ 2986.390625 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 881.26953125( disk./.usagePercent‚75.9343727131982 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 881.26953125 disk.sda.out‚ 896.7578125 ¹[2÷- _wandbruntime‚3810  _runtime‚3810ÇÅ*|ÇøÁ & lr-AdamW/pg1‚0.0006730478346200546 & lr-AdamW/pg2‚0.0006730478346200546 trainer/global_step‚3199 % _timestamp‚1.7885284270017385e+09 _runtime‚3810.537217406 _step‚258B¿ó)ÌùÆ & lr-AdamW/pg1‚0.0006730478346200546 & lr-AdamW/pg2‚0.0006730478346200546 trainer/global_step‚3199 % _timestamp‚1.7885284270017385e+09 _runtime‚3810.537217406 _step‚258‚Žèõ%2ú- _wandbruntime‚3810  _runtime‚3810ƒƒÎÉÕ%ûìO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3650983495168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59140514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19320654 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295832956449280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80804000 trainer/global_step‚3199 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299650547515392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299650303181312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147250425 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80777657 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3650983495168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268041787719168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271693699188224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262869152411659 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3637326001851 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59141149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80804000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777657 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3651893466112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3817346732032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3817567126016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66472770 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚3810.538359861 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39819862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39819860 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777657 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3651893466112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3817346732032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3817567126016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66472770 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚3810.538359861 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39819862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39819860 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268123406804992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39831927 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295923182153728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3653014939648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299741697961984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80802488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268123406804992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39831927 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295923182153728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3653014939648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299741697961984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80802488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570„ÂI{ž2€- _wandbruntime‚3811  _runtime‚3811ï¾€EZÔ /loss/FRt3/detection_loss‚1.1973144317453261 1loss/SRqqt4/detection_loss‚1.9686545659514034 /loss/FBt2/detection_loss‚2.4206648423121524 _runtime‚3811.488924754 /loss/FRt4/detection_loss‚2.1249905222991727 1loss/SRqqt3/detection_loss‚0.8931174708670506 /loss/FBt4/detection_loss‚2.4560590545336405 _step‚261 1loss/FBt2/assignment_loss‚0.10959007374283548 .loss/FRt4/assignment_loss‚3.23939983912234 2loss/SRqqt1/assignment_loss‚1.7283745630245124 0loss/SRqqt1/detection_loss‚2.599152145641191 1loss/SRqqt2/detection_loss‚2.2263266255981042 .loss/FRt1/detection_loss‚3.456735814872541 epoch‚2 $ _timestamp‚1.788528427953639e+09 .loss/FRt2/detection_loss‚2.194869501258275 /loss/FRt1/assignment_loss‚4.234493745000739 /loss/FBt1/detection_loss‚2.7108827395872637 trainer/global_step‚3199 .loss/FBt3/detection_loss‚0.692077871637814 %loss/total_loss‚44.25699229448817 0loss/FBt1/assignment_loss‚0.1948457170141906 2loss/FBt3/assignment_loss‚0.038305351790605444 1loss/FBt4/assignment_loss‚0.25000073849369076 0loss/FRt2/assignment_loss‚3.9067695424670266 0loss/FRt3/assignment_loss‚1.2952177422121167 2loss/SRqqt2/assignment_loss‚1.8541361270962577 1loss/SRqqt3/assignment_loss‚1.007673534344361 2loss/SRqqt4/assignment_loss‚1.4573397338755576_/.kù‚Ù 1loss/FBt2/assignment_loss‚0.10959007374283548 .loss/FRt4/assignment_loss‚3.23939983912234 2loss/SRqqt1/assignment_loss‚1.7283745630245124 0loss/SRqqt1/detection_loss‚2.599152145641191 1loss/SRqqt2/detection_loss‚2.2263266255981042 .loss/FRt1/detection_loss‚3.456735814872541 epoch‚2 $ _timestamp‚1.788528427953639e+09 .loss/FRt2/detection_loss‚2.194869501258275 /loss/FRt1/assignment_loss‚4.234493745000739 /loss/FBt1/detection_loss‚2.7108827395872637 6Tžftrainer/global_step‚3199 .loss/FBt3/detection_loss‚0.692077871637814 %loss/total_loss‚44.25699229448817 0loss/FBt1/assignment_loss‚0.1948457170141906 2loss/FBt3/assignment_loss‚0.038305351790605444 1loss/FBt4/assignment_loss‚0.25000073849369076 0loss/FRt2/assignment_loss‚3.9067695424670266 0loss/FRt3/assignment_loss‚1.2952177422121167 2loss/SRqqt2/assignment_loss‚1.8541361270962577 1loss/SRqqt3/assignment_loss‚1.007673534344361 2loss/SRqqt4/assignment_loss‚1.4573397338755576 /loss/FRt3/detection_loss‚1.1973144317453261 1loss/SRqqt4/detection_loss‚1.9686545659514034 /loss/FBt2/detection_loss‚2.4206648423121524 _runtime‚3811.488924754 /loss/FRt4/detection_loss‚2.1249905222991727 1loss/SRqqt3/detection_loss‚0.8931174708670506 /loss/FBt4/detection_loss‚2.4560590545336405 _step‚261…†™nU“ƒ‚(:ˆ ¹ŽëÔþÚ¨á gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä—_XM„‚(: ¹ŽëÔôâäá disk.md0.out‚ 882.53515625 disk.dm-0.out‚ 60759.8359375 disk.sda.out‚898.0849609375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.out‚898.0849609375 disk.dm-0.in‚0 disk.md0p1.out‚ 882.53515625 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚45072.68359375$ memory_percent‚95.98877429962158( proc.memory.availableMB‚ 657.19921875 network.sent‚ 7.38322e+06 network.recv‚ 1.387505e+06 disk.sda2.in‚128.625 disk.rbd2.in‚0 cpu‚25.18632644647451 disk.sdb.out‚898.0849609375 disk.sdb2.out‚898.0849609375 disk.sda1.in‚0 disk.nvme0n1.in‚0) disk./.usagePercent‚75.91962241489001$ disk./.usageGB‚2715.253433227539 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚15687.15234375 disk.rbd0.out‚ 1.404296875* proc.memory.percent‚18.227481842041016 disk.sda1.out‚0! proc.memory.rssMB‚ 2986.390625¹6 î’…‚(:‡ ÈŽëÔ؇¯à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ßK†‚(:À ÈŽëÔÔµžá disk.sda.out‚899.94482421875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 884.34375 disk.sda1.out‚0 disk.sda2.in‚128.625( proc.memory.availableMB‚ 657.17578125) disk./.usagePercent‚75.91935128381736 disk.sdb.out‚899.94482421875 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.dm-0.out‚60912.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 45225.171875! disk.sda2.out‚899.94482421875 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 disk.rbd0.in‚0 disk.sda.in‚128.625# disk.nvme1n1.out‚15688.10546875 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚899.94482421875 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875! proc.memory.rssMB‚ 2986.390625 cpu‚25.185740130987423 network.recv‚ 1.393316e+06 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0$ memory_percent‚95.98891735076904 network.sent‚ 7.480191e+06# disk./.usageGB‚2715.24373626709 disk.sdb1.out‚0 disk.md0.out‚ 884.34375Gî“¢’‡‚(:‡ ׎ëԚ˲à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÌ‚qLˆ‚(:Á ׎ëÔÅ ‰á disk.sda2.out‚901.4443359375 cpu‚25.185153000320128 disk.md0p1.out‚ 885.7734375 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.227481842041016 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚901.4443359375 disk.sdb2.in‚ 70.9609375 network.recv‚ 1.397266e+06) disk./.usagePercent‚75.92038493103416$ disk./.usageGB‚2715.280704498291# disk.nvme1n1.out‚15688.27734375 disk.md0.out‚ 885.7734375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚45424.77734375 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sdb.out‚901.4443359375 disk.sda.out‚901.4443359375$ memory_percent‚95.98832130432129' proc.memory.availableMB‚ 657.2734375! proc.memory.rssMB‚ 2986.390625 proc.cpu.threads‚17 network.sent‚ 7.485519e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.dm-0.out‚ 61113.0546875 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0°#"û2‰- _wandbruntime‚3867  _runtime‚3867•ƒÎÇŠÁ & lr-AdamW/pg1‚0.0006701452547039322 & lr-AdamW/pg2‚0.0006701452547039322 trainer/global_step‚3249 % _timestamp‚1.7885284842804303e+09 _runtime‚3867.815960446 _step‚262êÌû Ì‹Æ & lr-AdamW/pg1‚0.0006701452547039322 & lr-AdamW/pg2‚0.0006701452547039322 trainer/global_step‚3249 % _timestamp‚1.7885284842804303e+09 _runtime‚3867.815960446 _step‚262†,º‰$2Œ- _wandbruntime‚3867  _runtime‚3867[°òiò'ìO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 trainer/global_step‚3249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60035837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67480768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67480770 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67480768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67480770 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82048902 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3876352371712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263182706137008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885284842814782e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272122742010368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19612725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263182723176368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275830740904448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3707070920192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275830740904448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640Ý‘r÷'ŽñO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272122742010368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3707980891136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275829812930560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67480768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67480770 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67480768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67480770 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82048902 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3876352371712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263182706137008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885284842814782e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272122742010368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19612725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263182723176368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275830740904448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3707070920192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275830740904448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 trainer/global_step‚3249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60035837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60053581 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67500928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82073759 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82047390 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304312004877312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300434467376128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263261740770912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3708192380416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885284853726797e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272204379099136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19618404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304311756362240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82047390 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67500928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275912553476608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272204361096192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3709102364672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚3868.909186628 H:DeviceStatsMonitor—çXx.on_train_batch_end/allocation.all.freed‚ 149548318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266956969276357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275913481463808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300434491316224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚3249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617793 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 _step‚264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‡Çø'‘ûM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19618404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304311756362240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82047390 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67500928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275912553476608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272204361096192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3709102364672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚3868.909186628 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149548318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266956969276357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275913481463808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300434491316224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚3249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617793 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 _step‚264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60053581 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67500928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82073759 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82047390 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304312004877312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300434467376128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263261740770912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3708192380416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885284853726797e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272204379099136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ˆˆÃd2’- _wandbruntime‚3868  _runtime‚3868æw¢D\“Ö 2loss/FBt3/assignment_loss‚0.050828850633898706 .loss/FRt1/detection_loss‚3.398166697276266 /loss/FRt2/detection_loss‚2.7317441907452302 /loss/FRt3/detection_loss‚1.7884296397988995 1loss/FBt1/assignment_loss‚0.03969862222826729 1loss/FBt2/assignment_loss‚0.08614906552247703 .loss/FBt2/detection_loss‚2.808223717742496 /loss/FBt3/detection_loss‚0.6766392571851612 &loss/total_loss‚46.627229510391174 /loss/FRt4/assignment_loss‚3.872257751577041 1loss/SRqqt4/detection_loss‚2.1180793670090763 _step‚265 /loss/FRt4/detection_loss‚2.5921448670765934 1loss/SRqqt2/detection_loss‚2.2806088941938736 _runtime‚3868.910143552 /loss/FRt3/assignment_loss‚1.236680010954539 2loss/SRqqt1/assignment_loss‚1.3151365844976335 1loss/SRqqt1/detection_loss‚3.1408545545169284 1loss/SRqqt3/detection_loss‚1.0059265958165728 trainer/global_step‚3249 /loss/FRt1/assignment_loss‚3.974312418385556 2loss/SRqqt4/assignment_loss‚1.9892772039906546 1loss/FBt4/assignment_loss‚0.13167342688070172 epoch‚2 .loss/FBt4/detection_loss‚2.075546317630344 % _timestamp‚1.7885284853748422e+09 2loss/SRqqt3/assignment_loss‚0.8670960042349339 .loss/FBt1/detection_loss‚2.794967527190844 /loss/FRt2/assignment_loss‚4.117106947405585 1loss/SRqqt2/assignment_loss‚1.535680997897597óä„8a”Û /loss/FRt1/assignment_loss‚3.974312418385556 2loss/SRqqt4/assignment_loss‚1.9892772039906546 1loss/FBt4/assignment_loss‚0.13167342688070172 epoch‚2 .loss/FBt4/detection_loss‚2.075546317630344 % _timestamp‚1.7885284853748422e+09 2loss/SRqqt3/assignment_loss‚0.8670960042349339 .loss/FBt1/detection_loss‚2.794967527190844 /loss/FRt2/assignment_loss‚4.117106947405585 1loss/SRqqt2/assignment_loss‚1.535680997897597 2loss/FBt3/assignment_loss‚0.050828850633898706 .loss/FRt1/detection_loss‚3.398166697276266 /loss/FRt2/detection_loss‚2.7317441907452302 /loss/FRt3/detection_loss‚1.7884296397988995 1loss/FBt1/assignment_loss‚0.03969862222826729 1loss/FBt2/assignment_loss‚0.08614906552247703 .loss/FBt2/detection_loss‚2.808223717742496 /loss/FBt3/detection_loss‚0.6766392571851612 &loss/total_loss‚46.627229510391174 /loss/FRt4/assignment_loss‚3.872257751577041 1loss/SRqqt4/detection_loss‚2.1180793670090763 _step‚265 /loss/FRt4/detection_loss‚2.5921448670765934 1loss/SRqqt2/detection_loss‚2.2806088941938736 _runtime‚3868.910143552 /loss/FRt3/assignment_loss‚1.236680010954539 2loss/SRqqt1/assignment_loss‚1.3151365844976335 1loss/SRqqt1/detection_loss‚3.1408545545169284 1loss/SRqqt3/detection_loss‚1.0059265958165728 trainer/global_step‚3249‰o)Ÿë”•‚(:‰ æŽëÔÿž“á gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÚrMO–‚(:Ä æŽëÔ³òá! disk.sda2.out‚902.75830078125 network.recv‚ 1.401113e+06) disk./.usagePercent‚75.92741492670629 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚15688.64453125$ memory_percent‚95.99010944366455 network.sent‚ 7.50186e+06% disk./.usageGB‚2715.5321311950684 disk.sda.in‚128.625 disk.sda2.in‚128.625 disk.md0p1.out‚ 887.0234375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2986.390625 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 61254.40625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚902.75830078125 disk.rbd2.in‚0! disk.sdb2.out‚902.75830078125 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚45565.76171875 disk.md0.in‚ 199.5859375 disk.sda.out‚902.75830078125 disk.rbd0.in‚0 disk.md0.out‚ 887.0234375( proc.memory.availableMB‚ 656.98046875 cpu‚25.184492765443995 disk.nvme0n1.in‚0* proc.memory.percent‚18.227481842041016ßG^û”—‚(:‰ õŽëÔáò à gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʆ…þ=˜‚(:² õŽëÔõúîà disk.rbd2.in‚0 disk.sdb2.out‚ 904.73046875 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.183926562654616 disk.md0.out‚888.9375* proc.memory.percent‚18.227481842041016 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 network.sent‚ 7.598828e+06 network.recv‚ 1.406546e+06 disk.dm-0.in‚0! proc.memory.rssMB‚ 2986.390625! disk.nvme0n1.out‚ 45877.921875 disk.sda2.out‚ 904.73046875 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 904.73046875 disk.sda1.out‚0' proc.memory.availableMB‚ 657.4296875( disk./.usagePercent‚75.9344122842556 disk.md0.in‚ 199.5859375# disk.nvme1n1.out‚15688.80859375 disk.rbd0.out‚ 1.404296875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚888.9375% disk./.usageGB‚2715.7823905944824 disk.sda2.in‚128.625 disk.dm-0.out‚61566.73046875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 904.73046875$ memory_percent‚95.98736763000488¼¾þv”™‚(:‰ „ëÔ¤À’à gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž7ÏéKš‚(:À „ëÔúœ³á disk.dm-0.out‚ 61777.515625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚906.31591796875 disk.nvme1n1.in‚0 disk.md0p1.out‚ 890.47265625! proc.memory.rssMB‚ 2986.390625 disk.dm-0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875* proc.memory.percent‚18.227481842041016 network.recv‚ 1.410936e+06 disk.sdb1.out‚0 cpu‚25.1834074126915$ disk./.usageGB‚2715.781410217285 disk.sda.in‚128.625 disk.rbd2.in‚0! disk.nvme0n1.out‚ 46087.984375$ memory_percent‚95.98886966705322 disk.md0.in‚ 199.5859375 disk.md0.out‚ 890.47265625! disk.sdb2.out‚906.31591796875 disk.md0p1.in‚ 199.5859375! disk.sda2.out‚906.31591796875 network.sent‚ 7.604927e+06 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.nvme0n1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.93438487249885 disk.sda.out‚906.31591796875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme1n1.out‚ 15689.53125 disk.sda2.in‚128.625( proc.memory.availableMB‚ 657.18359375z`dó”›‚(:‰ “ëÔ‚ܲà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÕ„Jœ‚(:¿ “ëÔâ‚æà disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 cpu‚25.182780531455517 network.recv‚ 1.414875e+06 disk.md0p1.out‚891.875( proc.memory.availableMB‚ 656.93359375 network.sent‚ 7.610284e+06% disk./.usageGB‚2715.2540397644043 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚62040.23046875" disk.nvme0n1.out‚ 46350.5234375 disk.sda2.in‚128.625! proc.memory.rssMB‚ 2986.390625 disk.md0.out‚891.875 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚15689.70703125 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875* proc.memory.percent‚18.227481842041016) disk./.usagePercent‚75.91963937391462 disk.sda.in‚128.625 disk.rbd0.out‚ 1.404296875 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.nvme0n1.in‚0! disk.sda2.out‚907.78369140625 disk.sdb.out‚907.78369140625$ memory_percent‚95.99039554595947 proc.cpu.threads‚17! disk.sdb2.out‚907.78369140625 disk.rbd0.in‚0 disk.sda.out‚907.78369140625ª‘Ñw2- _wandbruntime‚3925  _runtime‚3925ÈeÐYÇžÁ _runtime‚3925.625769582 _step‚266 & lr-AdamW/pg1‚0.0006671792704780315 & lr-AdamW/pg2‚0.0006671792704780315 trainer/global_step‚3299 % _timestamp‚1.7885285420901768e+09¼õÌŸÆ % _timestamp‚1.7885285420901768e+09 _runtime‚3925.625769582 _step‚266 & lr-AdamW/pg1‚0.0006671792704780315 & lr-AdamW/pg2‚0.0006671792704780315 trainer/global_step‚3299Šî,{2 - _wandbruntime‚3925  _runtime‚3925Nç« ²%¡êO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763158345216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276203696301568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3764068316160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3764068316160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935125469696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276203714304512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚3925.626616555 _step‚267 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276203714304512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304855526894080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3934913464320 B927973888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚3925.626616555 _step‚267 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276203714304512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304855526894080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3934913464320 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267212603458752 % _timestamp‚1.7885285432030294e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83318661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68508928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308881829694464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3936077095936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚268 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308881829694464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3936077095936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚268 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267212603458752 % _timestamp‚1.7885285432030294e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83318661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68508928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804957767175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69496770 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69496768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69496770 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3819245770240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84538706 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84512363 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788528599701031e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84512363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271084482630768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3993535337472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41629610 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284104824336896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237171200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309366812116480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280284650592768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61822994 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3983.237079609 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3993772508672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185Ô¤g¨ö'¶ðO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280284650592768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61822994 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3983.237079609 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3993772508672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3819245770240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804957767175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69496770 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69496768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69496770 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3819245770240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84538706 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84512363 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788528599701031e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84512363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271084482630768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3993535337472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41629610 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284104824336896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237171200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309366812116480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚37259ñ@Ð2·- _wandbruntime‚3984  _runtime‚3984þÈ|¸öM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3820367230464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41641675 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313451738828800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3984.410334715 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69516928 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280366287681536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84537194 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84563563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280366287681536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885286008728938e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41641677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313451991538176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3820367230464 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821277214720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994929777152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284186636909056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69516930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20199178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3806075040453 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280366269678592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0EáO'¹ûM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821277214720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994929777152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284186636909056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69516930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20199178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3806075040453 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280366269678592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3820367230464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41641675 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313451738828800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3984.410334715 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69516928 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280366287681536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84537194 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84563563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280366287681536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885286008728938e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41641677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313451991538176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3820367230464 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144» ®2º- _wandbruntime‚3984  _runtime‚3984øü§Y»Ó /loss/FRt1/detection_loss‚2.9786961248942783 1loss/SRqqt1/assignment_loss‚2.074666891247034 1loss/FBt1/assignment_loss‚0.09996426001244847 trainer/global_step‚3349 1loss/SRqqt1/detection_loss‚2.2898618794149823 /loss/FBt1/detection_loss‚1.8081505141760175 2loss/SRqqt2/assignment_loss‚1.8487718532031232 /loss/FRt3/detection_loss‚1.5640651356501907 0loss/SRqqt2/detection_loss‚2.345911439621087 epoch‚2 % _timestamp‚1.7885286008762648e+09 _runtime‚3984.411723482 _step‚273 /loss/FBt2/assignment_loss‚0.402837646601256 0loss/FBt3/assignment_loss‚0.0170912903422911 /loss/FRt1/assignment_loss‚3.051289681877409 2loss/SRqqt3/assignment_loss‚0.8390357726580137 .loss/FBt3/detection_loss‚0.863551533928043 /loss/FRt4/detection_loss‚2.1451437303856498 %loss/total_loss‚42.98265521492414 /loss/FRt3/assignment_loss‚1.136966982308556 0loss/FRt4/assignment_loss‚3.7216367827307795 1loss/SRqqt4/assignment_loss‚1.619890189680614 /loss/FBt2/detection_loss‚3.0789988338947296 /loss/FBt4/detection_loss‚1.9889307824464946 0loss/FRt2/assignment_loss‚3.6638603380748203 1loss/FBt4/assignment_loss‚0.10014317081681139 .loss/FRt2/detection_loss‚2.403820126401515 1loss/SRqqt3/detection_loss‚1.0642454570715691 1loss/SRqqt4/detection_loss‚1.8751247974864222/T8D^¼Ø /loss/FRt1/detection_loss‚2.9786961248942783 1loss/SRqqt1/assignment_loss‚2.074666891247034 1loss/FBt1/assignment_loss‚0.09996426001244847 trainer/global_step‚3349 1loss/SRqqt1/detection_loss‚2.2898618794149823 /loss/FBt1/detection_loss‚1.8081505141760175 2loss/SRqqt2/assignment_loss‚1.8487718532031232 /loss/FRt3/detection_loss‚1.5640651356501907 0loss/SRqqt2/detection_loss‚2.345911439621087 epoch‚2 % _timestamp‚1.7885286008762648e+09 _runtime‚3984.411723482 _step‚273 /loss/FBt2/assignment_loss‚0.402837646601256 0loss/FBt3/assignment_loss‚0.0170912903422911 /loss/FRt1/assignment_loss‚3.051289681877409 2loss/SRqqt3/assignment_loss‚0.8390357726580137 .loss/FBt3/detection_loss‚0.863551533928043 /loss/FRt4/detection_loss‚2.1451437303856498 %loss/total_loss‚42.98265521492414 /loss/FRt3/assignment_loss‚1.136966982308556 0loss/FRt4/assignment_loss‚3.7216367827307795 1loss/SRqqt4/assignment_loss‚1.619890189680614 /loss/FBt2/detection_loss‚3.0789988338947296 /loss/FBt4/detection_loss‚1.9889307824464946 0loss/FRt2/assignment_loss‚3.6638603380748203 1loss/FBt4/assignment_loss‚0.10014317081681139 .loss/FRt2/detection_loss‚2.403820126401515 1loss/SRqqt3/detection_loss‚1.0642454570715691 1loss/SRqqt4/detection_loss‚1.8751247974864222‘óR({”½‚(:‰ ÞëÔø‚·á gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.067' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.76675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ°¾Q¾‚(:Æ ÞëÔ‚¬ªâ network.recv‚ 1.436101e+06$ disk./.usageGB‚2715.782356262207 disk.rbd0.in‚0 disk.nvme0n1.out‚ 47490.1875 disk.nvme1n1.in‚0* proc.memory.percent‚17.696714401245117 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 disk.md0.in‚ 199.5859375 network.sent‚ 7.738651e+06 disk.sda1.out‚0 disk.dm-0.out‚63181.28515625 disk.sdb.in‚ 70.9609375$ memory_percent‚95.46020030975342 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sda.out‚915.90869140625 disk.rbd2.in‚0 disk.sdb.out‚915.90869140625) disk./.usagePercent‚75.93441132431083 disk.md0p1.out‚ 899.67578125 disk.md0.out‚ 899.67578125 disk.dm-0.in‚0! disk.sdb2.out‚915.90869140625 cpu‚25.179905829553036 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sda.in‚128.625 disk.sdb1.in‚0! disk.sda2.out‚915.90869140625# disk.nvme1n1.out‚15691.09765625( proc.memory.availableMB‚ 743.80078125" proc.memory.rssMB‚ 2899.4296875`mÍ/”¿‚(:‰ íëÔ°š¬à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u^Ç7À‚(:¬ íëÔÖ›á$ disk./.usageGB‚2715.781337738037 disk.sda.out‚ 917.328125 disk.rbd2.in‚0# disk.nvme0n1.out‚47832.96484375" proc.memory.rssMB‚ 2899.4296875 network.sent‚ 7.845624e+06 disk.sdb2.out‚ 917.328125 disk.sda.in‚128.625 disk.dm-0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.440999e+06 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.404296875 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.dm-0.out‚ 63524.203125 disk.sdb.out‚ 917.328125# disk.nvme1n1.out‚15691.23828125 disk.md0.in‚ 199.5859375* proc.memory.percent‚17.696714401245117 cpu‚25.179374785913712 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 901.03125 disk.sdb.in‚ 70.9609375& proc.memory.availableMB‚ 744.203125 disk.rbd0.in‚0 disk.rbd2.out‚ 1.341796875) disk./.usagePercent‚75.93438284594875 disk.sda1.in‚0 disk.sda2.out‚ 917.328125 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 901.03125 disk.sdb1.in‚0 disk.sda1.out‚0$ memory_percent‚95.45774459838867ьĔÁ‚(:‰ üëԥПà gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥RiN‚(:à üëÔØùà network.recv‚ 1.445589e+06 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0" proc.memory.rssMB‚ 2899.4296875* proc.memory.percent‚17.696714401245117 disk.sdb.out‚918.60302734375 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.404296875 cpu‚25.178836482026167) disk./.usagePercent‚75.91964140046473 disk.rbd2.out‚ 1.341796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0# disk.nvme1n1.out‚15691.44921875 disk.sdb1.in‚0 network.sent‚ 7.853894e+06% disk./.usageGB‚2715.2541122436523! disk.nvme0n1.out‚ 48075.953125 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.md0p1.out‚ 902.2578125 disk.rbd2.in‚0! disk.sda2.out‚918.60302734375 disk.md0.out‚ 902.2578125! disk.sdb2.out‚918.60302734375$ proc.memory.availableMB‚744.1875 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.dm-0.out‚63767.40234375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sda.out‚918.60302734375$ memory_percent‚95.45783996582031ìD.”Â(:‰ ‹ëÔ¿¾˜à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fi MÄ‚(: ‹ëÔ•Èåà# memory_percent‚95.4578161239624 disk.dm-0.in‚0 disk.dm-0.out‚63953.65234375 disk.sdb1.out‚0! disk.sda2.out‚920.62646484375 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0* proc.memory.percent‚17.696714401245117 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 cpu‚25.178300700384092) disk./.usagePercent‚75.92112270193483# disk.nvme0n1.out‚48261.96484375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2899.4296875 proc.cpu.threads‚17% disk./.usageGB‚2715.3070907592773 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.md0.out‚ 904.203125! disk.sdb2.out‚920.62646484375 disk.rbd0.out‚ 1.404296875( proc.memory.availableMB‚ 744.19140625 network.sent‚ 7.859252e+06 disk.sdb.out‚920.62646484375 disk.md0p1.out‚ 904.203125 disk.rbd2.out‚ 1.341796875 network.recv‚ 1.449423e+06 disk.sdb2.in‚ 70.9609375 disk.sda.out‚920.62646484375 disk.nvme1n1.out‚ 15691.6875 disk.sdb1.in‚0v¶ð2Å- _wandbruntime‚4040  _runtime‚4040®ï4ÇÆÁ & lr-AdamW/pg2‚0.0006610595746081141 trainer/global_step‚3399 % _timestamp‚1.7885286572242584e+09 _runtime‚4040.759772884 _step‚274 & lr-AdamW/pg1‚0.0006610595746081141Æ×ÄÌÇÆ & lr-AdamW/pg2‚0.0006610595746081141 trainer/global_step‚3399 % _timestamp‚1.7885286572242584e+09 _runtime‚4040.759772884 _step‚274 & lr-AdamW/pg1‚0.0006610595746081141’ôðúŽ2È- _wandbruntime‚4040  _runtime‚4040º®w®ÉëO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156262033 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317930278923264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3875333195264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788528657225309e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275035345318608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284365622886912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85757265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317930533743104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70504770 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288240938079232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85783608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4052181584384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42232862 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288241866053120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3875333195264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313878121278976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚4040.760877608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70504768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20483997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861742997283 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3860835022283 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284365604883968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 trainer/global_step‚3399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70504768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278897105355251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ RÕÊWC3876243166208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20483363 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20483997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861742997283 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3860835022283 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284365604883968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 trainer/global_step‚3399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70504768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278897105355251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3876243166208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20483363 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284365622886912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85757265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317930533743104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70504770 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288240938079232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85783608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4052181584384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42232862 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288241866053120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3875333195264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859“PŒÖ2Ë- _wandbruntime‚4041  _runtime‚4041Þæ2ü&ÌöM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275114362913152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313968323043328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85808465 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284447241972736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288323678625280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288324606612480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70524928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚4041.860327177 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85782096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284447223969792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3876454655488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85808465 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85782096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318021679406592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4053578841088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚276 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156333395 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3877364639744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20489165 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053356363264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70524928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42244927 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42244925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278977240223281 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288323678625280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288324606612480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70524928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚4041.860327177 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85782096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284447223969792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3876454655488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85808465 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85782096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318021679406592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4053578841088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚276 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156333395 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3877364639744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20489165 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053356363264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70524928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42244927 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42244925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278977240223281 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85808465 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284447241972736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4111305366016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4097.994576034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚3449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292378907769344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292378907769344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71512770 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318389406501376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3916712277391 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87002167 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚3449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292378907769344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292378907769344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71512770 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318389406501376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3916712277391 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87002167 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4111305366016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4097.994576034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570—DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158586297 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279065259679712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20781800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3917829550669 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocat¢üA‡ed‚322592122774016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42848175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282983072191021 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788528715611976e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4099.148730914 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20782427 _step‚280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158559926 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87026998 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63630604 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3918737525877 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87053367 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158559926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71532928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322591882647552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944Q`äÔ'áúM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87053367 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158559926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71532928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322591882647552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158586297 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279065259679712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20781800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3917829550669 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322592122774016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42848175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282983072191021 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788528715611976e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4099.148730914 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20782427 _step‚280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158559926 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87026998 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63630604 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3918737525877 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0˜xÞÍœ2â- _wandbruntime‚4099  _runtime‚4099úÞ=)WãÑ .loss/FRt4/detection_loss‚2.257532481590043 1loss/SRqqt2/detection_loss‚2.3710176372528076 0loss/FBt4/assignment_loss‚0.5177252283479296 1loss/SRqqt1/detection_loss‚2.7086955581022347 /loss/FBt1/detection_loss‚2.5379613297326222 .loss/FBt2/detection_loss‚2.248730449115529 trainer/global_step‚3449 _step‚281 .loss/FRt3/assignment_loss‚1.25131613283015 2loss/SRqqt2/assignment_loss‚1.8544161772727967 1loss/FBt2/assignment_loss‚0.14596981485370258 /loss/FRt2/detection_loss‚2.7886599618941545 0loss/SRqqt4/detection_loss‚1.982638569040732 % _timestamp‚1.7885287156144125e+09 /loss/FRt1/detection_loss‚2.9471171220143635 /loss/FRt2/assignment_loss‚4.106765912845731 1loss/SRqqt3/detection_loss‚0.8457054606001628 /loss/FBt3/detection_loss‚0.7122645392150119 0loss/FRt4/assignment_loss‚3.0661263530669003 _runtime‚4099.149719408 0loss/FBt1/assignment_loss‚0.2048523929995229 /loss/FRt3/detection_loss‚1.2583415835206189 /loss/FBt4/detection_loss‚1.9091939012209573 $loss/total_loss‚44.0531794039438 epoch‚2 1loss/FBt3/assignment_loss‚0.08322962230528126 /loss/FRt1/assignment_loss‚4.058799799283346 1loss/SRqqt1/assignment_loss‚1.693066928859638 1loss/SRqqt3/assignment_loss‚0.983980151373697 2loss/SRqqt4/assignment_loss‚1.5190722966058687“+ÿi\äÖ 1loss/SRqqt1/detection_loss‚2.7086955581022347 /loss/FBt1/detection_loss‚2.5379613297326222 .loss/FBt2/detection_loss‚2.248730449115529 trainer/global_step‚3449 _step‚281 .loss/FRt3/assignment_loss‚1.25131613283015 2loss/SRqqt2/assignment_loss‚1.8544161772727967 1loss/FBt2/assignment_loss‚0.14596981485370258 /loss/FRt2/detection_loss‚2.7886599618941545 0loss/SRqqt4/detection_loss‚1.982638569040732 % _timestamp‚1.7885287156144125e+09 /loss/FRt1/detection_loss‚2.9471171220143635 /loss/FRt2/assignment_loss‚4.106765912845731 1loss/SRqqt3/detection_loss‚0.8457054606001628 /loss/FBt3/detection_loss‚0.7122645392150119 0loss/FRt4/assignment_loss‚3.0661263530669003 _runtime‚4099.149719408 0loss/FBt1/assignment_loss‚0.2048523929995229 /loss/FRt3/detection_loss‚1.2583415835206189 /loss/FBt4/detection_loss‚1.9091939012209573 $loss/total_loss‚44.0531794039438 epoch‚2 1loss/FBt3/assignment_loss‚0.08322962230528126 /loss/FRt1/assignment_loss‚4.058799799283346 1loss/SRqqt1/assignment_loss‚1.693066928859638 1loss/SRqqt3/assignment_loss‚0.983980151373697 2loss/SRqqt4/assignment_loss‚1.5190722966058687 .loss/FRt4/detection_loss‚2.257532481590043 1loss/SRqqt2/detection_loss‚2.3710176372528076 0loss/FBt4/assignment_loss‚0.5177252283479296™Ù°²a“å‚(:ˆ ÖëÔ¶âùà gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.098' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.2745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_;¾–Næ‚(:à ÖëÔûÃ’á disk.sda.in‚128.625 disk.sdb.out‚928.3505859375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.dm-0.in‚0" proc.memory.rssMB‚ 2899.4296875) disk./.usagePercent‚75.93438156602235# disk./.usageGB‚2715.78129196167 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 16180.3828125 disk.rbd2.out‚ 1.341796875 cpu‚25.175605761641012 disk.sdb2.out‚928.3505859375 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 70.9609375 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚65176.91015625 network.sent‚ 7.987598e+06 disk.rbd0.in‚0 disk.md0.out‚ 911.63671875$ memory_percent‚95.46008110046387 proc.cpu.threads‚17 disk.sda2.out‚928.3505859375 network.recv‚ 1.471046e+06 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚48996.52734375* proc.memory.percent‚17.696714401245117 disk.sda.out‚928.3505859375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 911.63671875' proc.memory.availableMB‚ 743.8203125<Ó¨”ç‚(:‰ åëÔÀˆ¯à gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g2©Eè‚(:º åëÔ¼‚á) disk./.usagePercent‚75.91963052109045# disk.nvme1n1.out‚16224.04296875 disk.md0p1.out‚ 912.98828125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚49206.17578125 disk.md0.in‚ 199.5859375 network.sent‚ 8.095269e+06% disk./.usageGB‚2715.2537231445312 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0.out‚ 912.98828125 disk.dm-0.in‚0 disk.dm-0.out‚ 65430.21875 cpu‚25.175150535341256 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 929.76953125 disk.nvme0n1.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375& proc.memory.availableMB‚ 743.546875" proc.memory.rssMB‚ 2899.4296875 disk.sdb2.out‚ 929.76953125 disk.sda.in‚128.625 disk.sda2.in‚128.625* proc.memory.percent‚17.696714401245117 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 929.76953125 disk.nvme1n1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚ 929.76953125$ memory_percent‚95.46175003051758 network.recv‚ 1.476653e+06 disk.sdb1.out‚0¤¹è’é‚(:‡ ôëÔÕ¢Ïá gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.95' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nöbEê‚(:º ôëÔô΢â disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 16224.8046875$ memory_percent‚95.45669555664062 proc.cpu.threads‚17% disk./.usageGB‚2715.2796363830566 disk.md0p1.out‚ 914.6953125" disk.nvme0n1.out‚ 49363.5234375# proc.memory.availableMB‚744.375" proc.memory.rssMB‚ 2899.4296875 disk.sda1.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.92035506608515 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sda2.out‚931.5361328125 disk.dm-0.in‚0 disk.dm-0.out‚ 65588.328125 network.recv‚ 1.481151e+06 disk.rbd0.out‚ 1.404296875 disk.sdb.out‚931.5361328125 disk.sda.out‚931.5361328125 disk.nvme1n1.in‚0 network.sent‚ 8.10372e+06 disk.sda1.in‚0 disk.md0.out‚ 914.6953125 disk.sdb2.out‚931.5361328125* proc.memory.percent‚17.696714401245117 cpu‚25.17444687623538 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375ƒ¡¦å2ë- _wandbruntime‚4155  _runtime‚4155Zì…°ÆìÀ _runtime‚4155.005183744 _step‚282 & lr-AdamW/pg1‚0.0006546938480863117 & lr-AdamW/pg2‚0.0006546938480863117 trainer/global_step‚3499 $ _timestamp‚1.788528771466343e+09WIËíÅ _step‚282 & lr-AdamW/pg1‚0.0006546938480863117 & lr-AdamW/pg2‚0.0006546938480863117 trainer/global_step‚3499 $ _timestamp‚1.788528771466343e+09 _runtime‚4155.005183744š8 :2î- _wandbruntime‚4155  _runtime‚4155Ù[7žïëO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296515949485568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚283 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292527531469312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3973497507499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322900691723776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4169977792512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327070645576192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788528771467168e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64508034 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286910619319867 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3972589532499 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88273412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88247069 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72520770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296515021511680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3987508045312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327070910881792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282937104773008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚4155.005296305N.üö'ððO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88247069 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72520770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296515021511680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3987508045312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327070910881792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282937104773008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚4155.005296305 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296515949485568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚283 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292527531469312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3973497507499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322900691723776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4169977792512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327070645576192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788528771467168e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64508034 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286910619319867 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3972589532499 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88273412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231›Ñ´í-”ñ‚(:‰ ƒ‘ëÔ»“ÁÞ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„“ÎRò‚(:Ç ƒ‘ëÔÄûëà disk.md0.in‚ 199.5859375" proc.memory.rssMB‚ 2899.4296875 disk.dm-0.in‚0" disk.nvme0n1.out‚ 49505.6015625 disk.sda2.in‚128.625 disk.md0.out‚ 916.06640625& proc.memory.availableMB‚ 744.109375) disk./.usagePercent‚75.92127927959599# disk.nvme1n1.out‚16224.99609375 disk.md0p1.out‚ 916.06640625 disk.sdb.out‚932.96728515625* proc.memory.percent‚17.696714401245117 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd2.in‚0% disk./.usageGB‚2715.3126907348633 disk.sda.out‚932.96728515625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0! disk.sda2.out‚932.96728515625$ memory_percent‚95.45831680297852 network.sent‚ 8.109147e+06 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 70.9609375 network.recv‚ 1.484826e+06 disk.sda.in‚128.625 disk.nvme0n1.in‚0 cpu‚25.17383815330838 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.dm-0.out‚65730.59765625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚932.96728515625 disk.sda1.out‚0 ^ê2ó- _wandbruntime‚4156  _runtime‚4156$Iwû&ôõM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292609150555136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72540930 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292609150555136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322990917428224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327162309008384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286989829173329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72540930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64526581 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 4156.03812894 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160812828 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3988629505536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72540928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283016122367552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327162064687616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72540928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171391580160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171171199488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3974614780985 % _timestamp‚1.7885287725005963e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88298269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3989539489792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283016139406912 _step‚284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3989539489792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296598690044928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88271900 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389MJdTs"õúM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72540928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171391580160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171171199488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3974614780985 % _timestamp‚1.7885287725005963e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88298269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3989539489792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283016139406912 _step‚284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3989539489792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296598690044928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88271900 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292609150555136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72540930 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292609150555136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322990917428224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327162309008384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286989829173329 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72540930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64526581 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 4156.03812894 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160812828 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3988629505536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.JÒÏÈpeak‚ 14135737856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72540928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283016122367552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327162064687616œ •õ52ö- _wandbruntime‚4156  _runtime‚41568µV÷Ð epoch‚2 $ _timestamp‚1.788528772504007e+09 1loss/FBt1/assignment_loss‚0.11221461647146498 /loss/FRt3/detection_loss‚1.4815839171642438 1loss/SRqqt2/detection_loss‚2.2678725481033326 /loss/FBt3/detection_loss‚0.6037234731885637 /loss/FRt1/assignment_loss‚3.868365240097046 0loss/FRt3/assignment_loss‚1.2981494488194585 .loss/FRt2/detection_loss‚2.581527275699497 .loss/FBt2/detection_loss‚2.124409329891205 0loss/FBt3/assignment_loss‚0.2997673668505662 1loss/SRqqt1/detection_loss‚2.5796656750497364 0loss/SRqqt4/detection_loss‚2.023677922998156 &loss/total_loss‚45.270327811465364 _step‚285 /loss/FRt2/assignment_loss‚4.164414606775556 2loss/SRqqt4/assignment_loss‚2.1270275094679425 1loss/FBt2/assignment_loss‚0.24634089811576132 0loss/FRt4/assignment_loss‚2.9873566392992363 1loss/SRqqt1/assignment_loss‚1.876599446648643 2loss/SRqqt2/assignment_loss‚1.6925735359545797 .loss/FBt4/detection_loss‚3.636276966995663 -loss/FRt1/detection_loss‚3.05797513961792 trainer/global_step‚3499 _runtime‚4156.039487076 1loss/SRqqt3/assignment_loss‚1.034041830211539 0loss/FBt4/assignment_loss‚0.1615082792442182 .loss/FRt4/detection_loss‚2.088016087403063 1loss/SRqqt3/detection_loss‚0.8545896891843189 /loss/FBt1/detection_loss‚2.1026503682136535 f#Ø[øÕ 0loss/SRqqt4/detection_loss‚2.023677922998156 &loss/total_loss‚45.270327811465364 _step‚285 /loss/FRt2/assignment_loss‚4.164414606775556 2loss/SRqqt4/assignment_loss‚2.1270275094679425 1loss/FBt2/assignment_loss‚0.24634089811576132 0loss/FRt4/assignment_loss‚2.9873566392992363 1loss/SRqqt1/assignment_loss‚1.876599446648643 2loss/SRqqt2/assignment_loss‚1.6925735359545797 .loss/FBt4/detection_loss‚3.636276966995663 -loss/FRt1/detection_loss‚3.05797513961792 trainer/global_step‚3499 _runtime‚4156.039487076 1loss/SRqqt3/assignment_loss‚1.034041830211539 0loss/FBt4/assignment_loss‚0.1615082792442182 .loss/FRt4/detection_loss‚2.088016087403063 1loss/SRqqt3/detection_loss‚0.8545896891843189 /loss/FBt1/detection_loss‚2.1026503682136535 epoch‚2 $ _timestamp‚1.788528772504007e+09 1loss/FBt1/assignment_loss‚0.11221461647146498 /loss/FRt3/detection_loss‚1.4815839171642438 1loss/SRqqt2/detection_loss‚2.2678725481033326 /loss/FBt3/detection_loss‚0.6037234731885637 /loss/FRt1/assignment_loss‚3.868365240097046 0loss/FRt3/assignment_loss‚1.2981494488194585 .loss/FRt2/detection_loss‚2.581527275699497 .loss/FBt2/detection_loss‚2.124409329891205 0loss/FBt3/assignment_loss‚0.2997673668505662 1loss/SRqqt1/detection_loss‚2.5796656750497364š+í”ù‚(:‰ ’‘ëÔÀ¹ß gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~NªTú‚(:É ’‘ëÔ«¸ìß disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sdb1.out‚0$ memory_percent‚95.45872211456299( proc.memory.availableMB‚ 744.04296875 disk.md0p1.out‚ 917.30078125* proc.memory.percent‚17.696714401245117 network.recv‚ 1.488658e+06 disk.md0.in‚ 199.5859375 disk.sda.out‚934.25537109375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875) disk./.usagePercent‚75.92738783493114$ disk./.usageGB‚2715.531162261963 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚49791.43359375 disk.sda2.in‚128.625 disk.md0.out‚ 917.30078125 disk.sdb.out‚934.25537109375 disk.md0p1.in‚ 199.5859375 network.sent‚ 8.112543e+06! disk.sda2.out‚934.25537109375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚934.25537109375 cpu‚25.173288584051082 disk.sda1.in‚0 disk.sda.in‚128.625 disk.sdb1.in‚0" proc.memory.rssMB‚ 2899.4296875 proc.cpu.threads‚17 disk.dm-0.out‚ 66016.7421875# disk.nvme1n1.out‚16225.30859375 disk.nvme1n1.in‚0 disk.dm-0.in‚0"iè”û‚(:‰ ¡‘ëÔÜýŒß gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´³ƒCü‚(:¸ ¡‘ëÔÐðÊà cpu‚25.172840156397495 disk.sda1.in‚0" disk.nvme0n1.out‚ 50115.9453125 disk.dm-0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625# proc.memory.rssMB‚ 2899.43359375) disk./.usagePercent‚75.93440481801836$ disk./.usageGB‚2715.782123565674 disk.rbd2.in‚0 disk.dm-0.out‚66341.42578125 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚ 936.30078125 disk.sda.out‚ 936.30078125# disk.nvme1n1.out‚16225.48046875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 936.30078125 disk.rbd0.in‚0( proc.memory.availableMB‚ 744.23828125 network.sent‚ 8.220391e+06 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 919.28125 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875* proc.memory.percent‚17.696738243103027 network.recv‚ 1.494322e+06 disk.sda1.out‚0 disk.sda2.out‚ 936.30078125 disk.md0p1.out‚ 919.28125 proc.cpu.threads‚17 disk.nvme1n1.in‚0$ memory_percent‚95.45753002166748li…Š”ý‚(:‰ °‘ëÔÕÖòÞ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t»~3Mþ‚(: °‘ëÔ™ùà* proc.memory.percent‚17.696738243103027! disk.nvme0n1.out‚ 50291.921875 disk.sda.out‚937.77392578125 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 920.6953125$ memory_percent‚95.45669555664062 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 920.6953125 disk.dm-0.in‚0 cpu‚25.172350275800195 network.sent‚ 8.226557e+06 disk.nvme0n1.in‚0 disk.dm-0.out‚66518.23046875) disk./.usagePercent‚75.93437228655606$ disk./.usageGB‚2715.780960083008 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.sdb.out‚937.77392578125 disk.sda.in‚128.625# disk.nvme1n1.out‚16226.30859375 network.recv‚ 1.498694e+06 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 2899.43359375 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚937.77392578125 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0! disk.sda2.out‚937.77392578125# proc.memory.availableMB‚744.375 proc.cpu.threads‚17G=¡;2ÿ- _wandbruntime‚4210  _runtime‚4210+9›˜Ç€Á _runtime‚4210.605026444 _step‚286 & lr-AdamW/pg1‚0.0006514203736132809 & lr-AdamW/pg2‚0.0006514203736132809 trainer/global_step‚3549 % _timestamp‚1.7885288270693748e+09‰Ž¦ÌÆ % _timestamp‚1.7885288270693748e+09 _runtime‚4210.605026444 _step‚286 & lr-AdamW/pg1‚0.0006514203736132809 & lr-AdamW/pg2‚0.0006514203736132809 trainer/global_step‚3549žè<2‚- _wandbruntime‚4210  _runtime‚4210ƽ“1ñ'ƒëO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286887984500208 % _timestamp‚1.7885288270705037e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163020739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4044505441280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296608467757568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 trainer/global_step‚3549 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4229119584256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89518314 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290917376302175 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290916451287815 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73528770 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4043595470336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚287 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300652063227904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44042610 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21361903 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚ 4210.60638526 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296608467757568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327411976946176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296608467757568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327411976946176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4043595470336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚287 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300652063227904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44042610 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21361903 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚ 4210.60638526 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286967019134112 % _timestamp‚1.7885288280879412e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300735731761152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44054677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331732229930496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163092101 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 D;DeviceStatsMonitor.onYVׄ™_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73548930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21367014 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230290475008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300735731761152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331732493125632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290997511170205 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263195136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73548928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44054675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163065730 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65422344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300734803773952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163065730 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89543171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296690104846336 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚12314‘#Y'‡úM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44054675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163065730 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65422344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300734803773952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163065730 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89543171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296690104846336 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327502178710528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296690086843392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73548930 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286967019134112 % _timestamp‚1.7885288280879412e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300735731761152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44054677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331732229930496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163092101 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73548930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21367014 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230290475008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300735731761152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331732493125632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290997511170205 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263195136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73548928 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 _step‚291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331923262168576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290838881266768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90736873 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294923208270123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885288829027739e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74536768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304790032918016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21652331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_r Nõeejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 _step‚291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331923262168576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290838881266768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90736873 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294923208270123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885288829027739e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74536768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304790032918016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21652331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚3599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44657925 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300771059137536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4101714339840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚332013487873024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300771059137536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788528884002488e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290917898861312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336302691811840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4100804355584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640^´ûsV"›úM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290917898861312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336302691811840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4100804355584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74556930 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300771041134592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44657927 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304872773477376 _runtime‚4267.538993599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74556928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165345003 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165318632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚3599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44657925 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300771059137536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_poo.¯7«ªl.allocated‚ 4101714339840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚332013487873024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300771059137536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788528884002488e+09¤Ì@mð2œ- _wandbruntime‚4267  _runtime‚4267~5ðe]× /loss/FRt2/detection_loss‚1.9145337796655528 .loss/FRt4/detection_loss‚2.075660598106109 1loss/SRqqt1/detection_loss‚2.5137007942906133 1loss/SRqqt2/detection_loss‚2.5228260278701784 /loss/FBt1/detection_loss‚2.7785365906628696 trainer/global_step‚3599 .loss/FRt3/detection_loss‚1.455444448062631 $ _timestamp‚1.788528884004853e+09 2loss/FBt1/assignment_loss‚0.023847839324472177 /loss/FBt3/detection_loss‚0.6827419294338477 _runtime‚4267.540289035 2loss/SRqqt3/assignment_loss‚0.8831104420839924 _step‚293 /loss/FRt4/assignment_loss‚3.063310130284383 1loss/SRqqt1/assignment_loss‚1.942897477083736 2loss/SRqqt4/assignment_loss‚1.6059671404731015 .loss/FBt4/detection_loss‚2.805651493370533 epoch‚2 0loss/FBt4/assignment_loss‚0.5030442712595686 /loss/FBt2/detection_loss‚2.6658076643943787 2loss/SRqqt2/assignment_loss‚1.8795853627224763 1loss/FBt2/assignment_loss‚0.25242186611285433 1loss/FBt3/assignment_loss‚0.10956555021343374 /loss/FRt1/detection_loss‚2.9367276500253117 1loss/SRqqt4/detection_loss‚2.1333055016787155 0loss/FRt1/assignment_loss‚3.7109371178290425 /loss/FRt2/assignment_loss‚4.333763262158946 0loss/FRt3/assignment_loss‚1.2558590437545152 1loss/SRqqt3/detection_loss‚0.8335192419894754 &loss/total_loss‚44.882765222850736éâêbžÜ 0loss/FBt4/assignment_loss‚0.5030442712595686 /loss/FBt2/detection_loss‚2.6658076643943787 2loss/SRqqt2/assignment_loss‚1.8795853627224763 1loss/FBt2/assignment_loss‚0.25242186611285433 1loss/FBt3/assignment_loss‚0.10956555021343374 /loss/FRt1/detection_loss‚2.9367276500253117 1loss/SRqqt4/detection_loss‚2.1333055016787155 0loss/FRt1/assignment_loss‚3.7109371178290425 /loss/FRt2/assignment_loss‚4.333763262158946 0loss/FRt3/assignment_loss‚1.2558590437545152 1loss/SRqqt3/detection_loss‚0.8335192419894754 &loss/total_loss‚44.882765222850736 /loss/FRt2/detection_loss‚1.9145337796655528 .loss/FRt4/detection_loss‚2.075660598106109 1loss/SRqqt1/detection_loss‚2.5137007942906133 1loss/SRqqt2/detection_loss‚2.5228260278701784 /loss/FBt1/detection_loss‚2.7785365906628696 trainer/global_step‚3599 .loss/FRt3/detection_loss‚1.455444448062631 $ _timestamp‚1.788528884004853e+09 2loss/FBt1/assignment_loss‚0.023847839324472177 /loss/FBt3/detection_loss‚0.6827419294338477 _runtime‚4267.540289035 2loss/SRqqt3/assignment_loss‚0.8831104420839924 _step‚293 /loss/FRt4/assignment_loss‚3.063310130284383 1loss/SRqqt1/assignment_loss‚1.942897477083736 2loss/SRqqt4/assignment_loss‚1.6059671404731015 .loss/FBt4/detection_loss‚2.805651493370533 epoch‚2¥ÛrB”Ÿ‚(:‰ û‘ëÔÆÞÎß gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.545' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.13625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j­KH ‚(:½ û‘ëÔæ€ïß% disk./.usageGB‚2715.5318489074707 disk.sda1.in‚0 disk.md0.out‚ 927.84375 disk.sdb.in‚ 70.9609375 cpu‚25.169928147400864) disk./.usagePercent‚75.92740703382692# disk.nvme1n1.out‚16227.68359375 disk.sdb1.in‚0! disk.sda2.out‚945.22314453125 disk.dm-0.in‚0 disk.sdb.out‚945.22314453125 disk.sda.in‚128.625 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 67573.875 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 network.sent‚ 8.355796e+06 disk.nvme0n1.in‚0! disk.sdb2.out‚945.22314453125 network.recv‚ 1.519779e+06 disk.sda.out‚945.22314453125 disk.rbd0.in‚0* proc.memory.percent‚17.696738243103027 disk.nvme1n1.in‚0 disk.sda2.in‚128.625 disk.sda1.out‚0 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚51346.19140625# memory_percent‚95.4568862915039% proc.memory.availableMB‚ 744.34375# proc.memory.rssMB‚ 2899.43359375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 927.84375 disk.sdb1.out‚0ŸƒÒ¹”¡‚(:‰ Š’ëÔêà¹à gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú²ÑÐQ¢‚(:Æ Š’ëÔò±îà network.sent‚ 8.462876e+06" disk.nvme0n1.out‚ 51632.6796875 disk.md0.in‚ 199.5859375 disk.sdb.out‚947.15185546875 disk.nvme1n1.out‚ 16227.8125 cpu‚25.169498997975168 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sda.out‚947.15185546875( proc.memory.availableMB‚ 744.36328125 disk.sdb1.in‚0$ memory_percent‚95.45676708221436 network.recv‚ 1.524589e+06 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 disk.md0.out‚ 929.70703125 disk.md0p1.out‚ 929.70703125# proc.memory.rssMB‚ 2899.43359375* proc.memory.percent‚17.696738243103027 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0! disk.sda2.out‚947.15185546875 disk.dm-0.in‚0 disk.dm-0.out‚ 67860.4921875) disk./.usagePercent‚75.93439873836803% disk./.usageGB‚2715.7819061279297 disk.sdb1.out‚0! disk.sdb2.out‚947.15185546875 disk.sda.in‚128.625 disk.nvme1n1.in‚0;).î’£‚(:‡ ™’ëÔ…™žà gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rì‡ÿE¤‚(:º ™’ëÔõôÑà disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.dm-0.in‚0 disk.sdb.out‚948.51904296875 disk.sdb2.in‚ 70.9609375 disk.sda2.in‚128.625" proc.memory.rssMB‚ 2984.3359375* proc.memory.percent‚18.214941024780273( disk./.usagePercent‚75.9196555863155 disk.nvme1n1.in‚0! disk.sda2.out‚948.51904296875 disk.md0.out‚ 931.015625 cpu‚25.176017724027325 network.sent‚ 8.471404e+06 disk.sdb1.in‚0 disk.sda1.out‚0! proc.memory.availableMB‚659.5 disk.rbd0.out‚ 1.404296875# memory_percent‚95.9747314453125 disk.sda.in‚128.625 disk.md0p1.out‚ 931.015625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 51938.7578125 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 68166.7421875! disk.sdb2.out‚948.51904296875 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.in‚0 network.recv‚ 1.529525e+06% disk./.usageGB‚2715.2546195983887 disk.sdb.in‚ 70.9609375 disk.sda.out‚948.51904296875! disk.nvme1n1.out‚ 16227.984375 proc.cpu.threads‚17aÏŽ”¥‚(:‰ ¨’ëÔ‰ž¾ä gpu.0.gpu‚62.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.867' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.96675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¥DM¦‚(: ¨’ëÔñÀæ network.recv‚ 1.533945e+06 disk.md0.in‚ 199.5859375 disk.sda.out‚949.82177734375 disk.md0p1.out‚ 932.265625 disk.sda1.in‚0 network.sent‚ 8.475347e+06" disk.nvme0n1.out‚ 52099.1015625 disk.dm-0.out‚68327.48828125 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.91839976521031 disk.md0.out‚ 932.265625 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.nvme0n1.in‚0! disk.sda2.out‚949.82177734375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚949.82177734375 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875& proc.memory.availableMB‚ 659.390625 cpu‚25.19001434005395 disk.sda2.in‚128.625 disk.sdb.out‚949.82177734375# disk.nvme1n1.out‚16228.38671875$ disk./.usageGB‚2715.209705352783 disk.dm-0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0" proc.memory.rssMB‚ 2984.3359375* proc.memory.percent‚18.214941024780273 disk.sdb1.in‚0$ memory_percent‚95.97539901733398po.¥’§‚(:‡ ·’ëÔÕ€ãà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:yä`J¨‚(:¿ ·’ëÔ›ÿà disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0* proc.memory.percent‚18.214941024780273% disk./.usageGB‚2715.2942390441895" disk.nvme1n1.out‚ 16228.6015625! disk.sda2.out‚951.58837890625" proc.memory.rssMB‚ 2984.3359375 disk.nvme0n1.in‚0 proc.cpu.threads‚17! disk.nvme0n1.out‚ 52192.703125 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚951.58447265625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 933.97265625 disk.sda2.in‚128.625 network.recv‚ 1.537913e+06 disk.md0.in‚ 199.5859375 disk.sda.out‚951.58837890625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.rbd2.out‚ 1.341796875) disk./.usagePercent‚75.92076336260212 disk.md0.out‚ 933.96875 disk.sdb.out‚951.57666015625 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 cpu‚25.20474044572399 disk.dm-0.out‚ 68421.3046875 disk.sda1.out‚0# memory_percent‚95.9756851196289% proc.memory.availableMB‚ 659.34375 network.sent‚ 8.482865e+068hsK”©‚(:‰ Æ’ëÔ¿¬®à gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝUñLª‚(:Á Æ’ëÔÞì“á* proc.memory.percent‚18.214941024780273 disk.sdb1.out‚0 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.sda2.out‚953.1826171875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚953.1826171875# disk.nvme1n1.out‚16314.98828125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 cpu‚25.21855399533164 network.sent‚ 8.488634e+06( disk./.usagePercent‚75.9208297054531% disk./.usageGB‚2715.2966117858887( proc.memory.availableMB‚ 659.34765625" proc.memory.rssMB‚ 2984.3359375 disk.dm-0.out‚ 68665.1328125 disk.rbd2.in‚0 disk.rbd0.out‚ 1.404296875 network.recv‚ 1.54196e+06# disk.nvme0n1.out‚52350.14453125 disk.md0.out‚ 935.51171875 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda2.in‚128.625 disk.sdb2.out‚953.1826171875 disk.rbd2.out‚ 1.341796875 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.md0p1.out‚ 935.51171875 disk.nvme0n1.in‚0 disk.sda.out‚953.1826171875 disk.sda1.in‚0" memory_percent‚95.975661277771&DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚295522253068016 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91890843 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4139797838567 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚309789575140352 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75859574 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚337206668088832 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚75859572 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚295522253068016 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91890843 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4139797838567 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚309789575140352 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75859574 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚337206668088832 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚75859572 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 Fctive_split_bytes.small_pool.peak‚ 353397248 % _timestamp‚1.7885289735613506e+09 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4140478906011 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚305633540895232 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135737856 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91870133 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚309788874098688 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4156016242176 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038720 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038720 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167729705 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4345360274944 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570¦ Èäí2®- _wandbruntime‚4358  _runtime‚4358Ó*Ô»c)¯ÝR [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚45461782 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚295 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚309843624856576 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038720 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167774707 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416517 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4140967116895 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32005 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041664 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚295572679173200 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4140286049451 % _timestamp‚1.7885289746723113e+09 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4155823054336 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4156506093056 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚3626 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚341608615266304 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚91899657 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚299712948183291 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚305687819805184 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚22088258 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167753995 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67550040 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91878947 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4155823054336 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚75875050 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 194442240 _runtime‚4358.208264537 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167753995 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚309844325898240 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚295572662133840 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚305687801802240 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚305687801802240 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚222 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚299713646290095 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20710 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4345850125824 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106804 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚22088036 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767680 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚75875050 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67549816 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚45461780 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚337262935642624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4345679623680þwùh)°âR YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚309844325898240 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚295572662133840 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚305687801802240 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚305687801802240 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚222 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚299713646290095 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20710 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4345850125824 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106804 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚22088036 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767680 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚75875050 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67549816 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚45461780 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚337262935642624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4345679623680 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚45461782 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚295 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚309843624856576 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038720 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167774707 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416517 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4140967116895 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32005 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041664 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚295572679173200 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4140286049451 % _timestamp‚1.7885289746723113e+09 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4155823054336 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4156506093056 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚3626 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚341608615266304 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚91899657 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚299712948183291 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚305687819805184 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚22088258 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167753995 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67550040 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91878947 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4155823054336 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚75875050 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 194442240 _runtime‚4358.208264537 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167753995§ö’±‚(:‡ Õ’ëÔëÚŸà gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñsýS²‚(:È Õ’ëÔÚ—‚ã) disk./.usagePercent‚75.92258075140894 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda1.in‚0 disk.md0.out‚ 936.94140625 network.recv‚ 1.546629e+06 disk.md0p1.out‚ 936.94140625 disk.rbd0.in‚0 disk.sda1.out‚0% disk./.usageGB‚2715.3592376708984 disk.sda.out‚954.67138671875 disk.rbd2.out‚ 1.341796875* proc.memory.percent‚18.214941024780273! disk.sdb2.out‚954.67138671875" proc.memory.rssMB‚ 2984.3359375 disk.sdb1.in‚0# disk.nvme0n1.out‚52559.79296875& proc.memory.availableMB‚ 659.453125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 16317.9140625 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375$ memory_percent‚95.97501754760742 disk.sdb.out‚954.67138671875 disk.sdb1.out‚0! disk.sda2.out‚954.67138671875 proc.cpu.threads‚17 network.sent‚ 8.505415e+06 disk.dm-0.out‚68877.70703125 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0 cpu‚25.232765587590634úÁEx2³- _wandbruntime‚4373  _runtime‚4373íY$v´š§ ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23181632161140442 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899994134902954 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075753331184387 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4660087525844574 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870375216007233 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5125818848609924 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22660061717033386 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15432098507881165 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07586245983839035 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21020011603832245 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 jet/accuracy_2_of_7‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26420408487319946 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_2_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14569620788097382 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5459770560264587 +jet/accuracy_3_of_5‚0.12000768631696701 particle/accuracy_3_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24022988975048065 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.42359459400177 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15044279396533966 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15821225941181183 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17344939708709717 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1078282818198204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991459846496582 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4312312602996826 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_2‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2682062089443207 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5002052187919617 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3977071940898895 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7673711180686951 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1666661649942398 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5674293041229248 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15853174030780792 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03453454375267029 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35733553767204285 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2109636515378952 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4464735984802246 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8151999711990356 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17344939708709717 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4116428792476654 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03735632449388504 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6020833253860474 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1494118869304657 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5037485957145691 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17892460525035858 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/1FRt4SRqqtÆ ­+u*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904883503913879 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08395624160766602 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16352201998233795 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5549709796905518 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17333334684371948 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5026788711547852 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5595558285713196 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5632424354553223 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1979532092809677 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5455834865570068 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.42365968227386475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5191640257835388 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992361307144165 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10603829473257065 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015200003050267696 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 trainer/global_step‚3626 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1947280466556549 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8560164570808411 +particle/specificity‚0.9576572179794312 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.542714536190033 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3085080087184906 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01989247463643551 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16056330502033234 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 'particle/f_score‚0.3992865979671478 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.39421346783638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2680961489677429 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5568369626998901 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008934998884797096 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44476744532585144 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.49586036801338196 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16984300315380096 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823604226112366 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 +jet/accuracy_3_of_3‚0.03150457516312599 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19432374835014343 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7064108848571777 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35512983798980713 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875177145004272 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962656497955322 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5390902161598206 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 _step‚296 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 jet/accuracy_7_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.023167043924331665 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4285283088684082 jet/accuracy_6_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25901007652282715 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972724318504333 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4973774254322052 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5002466440200806 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 _runtime‚4373.970795218 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.525743842124939 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5260363817214966 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2268257886171341 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5398191809654236 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_2_of_2‚1 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3511904776096344 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988530278205872 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4279693365097046 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20353706181049347 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019315188750624657 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5304688215255737 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37522321939468384 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.986281156539917 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18039396405220032 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980880022049 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41855689883232117 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5111991167068481 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9985569715499878 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9906229376792908 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16782347857952118 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11918143182992935 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1805112510919571 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16286756098270416 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004007529001682997 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03230707347393036 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5192938446998596 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49497127532958984 particle/accuracy_6_of_6‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904730319976807 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5280789136886597 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7651515603065491 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007474403362721205 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5260363817214966 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611342906952 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2789040803909302 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 )loss/val_total_loss‚43.94035146675545 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03867203742265701 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31811580061912537 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46877819299697876 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_3‚0.8804246783256531 *jet/accuracy_1_of_4‚0.9246969223022461 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.055337391793727875 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.39877575635910034 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_7‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5078707933425903 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2672805190086365 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0036375662311911583 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861820340156555 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16744408011436462 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5624229907989502 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34407714009284973 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 jet/accuracy_5_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2655131220817566 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_7_of_7‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4990381896495819 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08175791800022125 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880883097648621 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640607833862305 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11644144356250763 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16960911452770233 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16123950481414795 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03938649594783783 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.027816718444228172 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.646965503692627 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5283717513084412 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.550168514251709 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22813786566257477 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5843807458877563 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.968849241733551 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16293461620807648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975408911705017 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43468019366264343 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5151865482330322 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40925922989845276 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993800163269043 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02646353654563427 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07520406693220139 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988530278205872 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20248396694660187 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44772785902023315 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4943556785583496 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6063601970672607 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5720395445823669 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990834653377533 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05112309381365776 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5653730630874634 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5734126567840576 +particle/sensitivity‚0.3066629469394684 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11550186574459076 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3729166090488434 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 % _timestamp‚1.7885289904331565e+09 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48679888248443604 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1950949728488922 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5945636034011841 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20574712753295898 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4802992343902588 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2478155493736267 particle/accuracy_1_of_3‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_6‚0.41025641560554504 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40511980652809143 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_5‚1 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11224335432052612 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6083332896232605 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 particle/accuracy_5_of_6‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9807027578353882 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46538347005844116 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5479143261909485 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4241078197956085 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5191888213157654 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1802179515361786 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34603095054626465 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.526849091053009 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18071715533733368 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915960431098938 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5907050967216492 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40867507457733154 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.317628413438797 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 particle/accuracy_3_of_3‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2699390649795532 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909277558326721 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47081655263900757 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5509857535362244 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22718918323516846 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_4_of_5‚0.02361111342906952 particle/accuracy_2_of_3‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 epoch‚2 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900476336479187 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4141325354576111 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08809523284435272 *jet/accuracy_1_of_5‚0.9590566158294678 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0²p‘Ç µŸ§ :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41855689883232117 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5111991167068481 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9985569715499878 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9906229376792908 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16782347857952118 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11918143182992935 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1805112510919571 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16286756098270416 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004007529001682997 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03230707347393036 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5192938446998596 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49497127532958984 particle/accuracy_6_of_6‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904730319976807 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5280789136886597 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.765151ÄäÇúù5603065491 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007474403362721205 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5260363817214966 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611342906952 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2789040803909302 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 )loss/val_total_loss‚43.94035146675545 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03867203742265701 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31811580061912537 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46877819299697876 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_3‚0.8804246783256531 *jet/accuracy_1_of_4‚0.9246969223022461 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.055337391793727875 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.39877575635910034 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_7‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5078707933425903 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2672805190086365 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0036375662311911583 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861820340156555 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16744408011436462 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5624229907989502 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34407714009284973 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 jet/accuracy_5_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2655131220817566 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_7_of_7‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4990381896495819 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08175791800022125 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880883097648621 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640607833862305 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11644144356250763 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16960911452770233 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16123950481414795 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03938649594783783 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.027816718444228172 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.646965503692627 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5283717513084412 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.550168514251709 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22813786566257477 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5843807458877563 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.968849241733551 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16293461620807648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975408911705017 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43468019366264343 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5151865482330322 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40925922989845276 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993800163269043 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02646353654563427 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07520406693220139 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988530278205872 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20248396694660187 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44772785902023315 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4943556785583496 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6063601970672607 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5720395445823669 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990834653377533 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05112309381365776 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5653730630874634 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5734126567840576 +particle/sensitivity‚0.3066629469394684 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11550186574459076 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3729166090488434 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 % _timestamp‚1.7885289904331565e+09 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48679888248443604 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1950949728488922 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5945636034011841 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20574712753295898 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4802992343902588 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2478155493736267 particle/accuracy_1_of_3‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_6‚0.41025641560554504 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40511980652809143 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_5‚1 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11224335432052612 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6083332896232605 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 particle/accuracy_5_of_6‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9807027578353882 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46538347005844116 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5479143261909485 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4241078197956085 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5191888213157654 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1802179515361786 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34603095054626465 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.526849091053009 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18071715533733368 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915960431098938 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5907050967216492 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40867507457733154 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.317628413438797 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 particle/accuracy_3_of_3‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2699390649795532 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909277558326721 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47081655263900757 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5509857535362244 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22718918323516846 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_4_of_5‚0.02361111342906952 particle/accuracy_2_of_3‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 epoch‚2 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900476336479187 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4141325354576111 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08809523284435272 *jet/accuracy_1_of_5‚0.9590566158294678 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23181632161140442 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899994134902954 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075753331184387 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4660087525844574 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870375216007233 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5125818848609924 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22660061717033386 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15432098507881165 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07586245983839035 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21020011603832245 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 jet/accuracy_2_of_7‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26420408487319946 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_2_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14569620788097382 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5459770560264587 +jet/accuracy_3_of_5‚0.12000768631696701 particle/accuracy_3_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24022988975048065 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.42359459400177 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15044279396533966 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15821225941181183 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17344939708709717 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1078282818198204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991459846496582 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4312312602996826 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_2‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2682062089443207 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5002052187919617 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3977071940898895 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7673711180686951 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1666661649942398 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5674293041229248 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15853174030780792 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03453454375267029 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35733553767204285 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2109636515378952 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4464735984802246 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8151999711990356 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17344939708709717 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4116428792476654 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03735632449388504 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6020833253860474 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1494118869304657 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5037485957145691 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17892460525035858 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904883503913879 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08395624160766602 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16352201998233795 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5549709796905518 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17333334684371948 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5026788711547852 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5595558285713196 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5632424354553223 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1979532092809677 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5455834865570068 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.42365968227386475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5191640257835388 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992361307144165 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10603829473257065 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015200003050267696 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 trainer/global_step‚3626 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1947280466556549 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8560164570808411 +particle/specificity‚0.9576572179794312 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.542714536190033 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3085080087184906 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01989247463643551 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16056330502033234 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 'particle/f_score‚0.3992865979671478 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.39421346783638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2680961489677429 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5568369626998901 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008934998884797096 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44476744532585144 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.49586036801338196 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16984300315380096 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823604226112366 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 +jet/accuracy_3_of_3‚0.03150457516312599 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19432374835014343 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7064108848571777 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35512983798980713 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875177145004272 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962656497955322 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5390902161598206 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 _step‚296 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 jet/accuracy_7_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.023167043924331665 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4285283088684082 jet/accuracy_6_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25901007652282715 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*öí=½æSRqqt1FBt/FBt_purity‚0.9972724318504333 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4973774254322052 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5002466440200806 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 _runtime‚4373.970795218 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.525743842124939 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5260363817214966 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2268257886171341 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5398191809654236 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_2_of_2‚1 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3511904776096344 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988530278205872 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4279693365097046 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20353706181049347 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019315188750624657 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5304688215255737 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37522321939468384 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.986281156539917 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18039396405220032 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980880022049¨½œÎ6¶‚(:… ä’ëÔÆßäÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.17' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.5425 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H˜‡N·‚(:à ä’ëÔ”ϧá disk.sda1.in‚0 disk.rbd2.out‚ 1.341796875 disk.rbd2.in‚0 disk.nvme0n1.in‚0& proc.memory.availableMB‚ 1.00390625 proc.cpu.threads‚17 network.recv‚ 1.552044e+06) disk./.usagePercent‚75.97027443498929% disk./.usageGB‚2717.0649948120117 disk.sda.out‚956.19775390625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 53047.2109375 disk.md0.out‚ 938.40234375! disk.sdb2.out‚956.19775390625" disk.nvme1n1.out‚ 16383.3828125 disk.md0p1.out‚ 938.40234375 disk.rbd0.in‚0 cpu‚25.240394693961843 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 disk.sda2.in‚128.625$ memory_percent‚99.99387264251709 disk.nvme1n1.in‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sdb.out‚956.19775390625 network.sent‚ 8.59998e+06 disk.sda1.out‚0 disk.dm-0.out‚ 69430.21875 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 3641.51953125) proc.memory.percent‚22.22607135772705! disk.sda2.out‚956.19775390625 disk.dm-0.in‚0‰ \“¸‚(:ˆ ó’ëÔɶà gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}„S¹‚(:È ó’ëÔüÒá' proc.memory.availableMB‚ 1319.796875" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.94573707291417 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.sdb.out‚957.81884765625 cpu‚25.238732385959278 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.190526962280273! disk.sda2.out‚957.81884765625 disk.dm-0.in‚0 disk.dm-0.out‚70583.26171875 disk.sdb2.in‚ 70.9609375 disk.sdb.in‚ 70.9609375 network.recv‚ 1.55718e+06" disk.nvme1n1.out‚ 16448.6640625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 939.96484375 disk.sda1.out‚0 disk.md0.in‚ 199.5859375$ memory_percent‚91.94459915161133 disk.sdb1.out‚0 disk.nvme0n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2716.1874198913574 disk.sda.out‚957.81884765625 disk.sda2.in‚128.625 disk.rbd2.in‚0# disk.nvme0n1.out‚54134.59765625 disk.md0.out‚ 939.96484375 network.sent‚ 8.722624e+06 disk.rbd2.out‚ 1.341796875! disk.sdb2.out‚957.81884765625ˆêo2º- _wandbruntime‚4404  _runtime‚4404PoÇ»Á & lr-AdamW/pg1‚0.0006446956130366617 & lr-AdamW/pg2‚0.0006446956130366617 trainer/global_step‚3649 % _timestamp‚1.7885290210131562e+09 _runtime‚4404.548684501 _step‚297ÌŒ‰Ì¼Æ & lr-AdamW/pg1‚0.0006446956130366617 & lr-AdamW/pg2‚0.0006446956130366617 trainer/global_step‚3649 % _timestamp‚1.7885290210131562e+09 _runtime‚4404.548684501 _step‚297©fI€u2½- _wandbruntime‚4404  _runtime‚4404nå>§ò'¾ìO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187557886976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76519758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45851516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45851518 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377959632896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92565336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4186647915520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885290210142148e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312376389371392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308188813481472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297966605620560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344357338972672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚4404.549839067 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187557886976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312376389371392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68135331 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302137584272088ÄZ—‹÷'¿ñO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187557886976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76519758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45851516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45851518 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377959632896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92565336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4186647915520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885290210142148e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312376389371392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308188813481472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297966605620560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344357338972672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚4404.549839067 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187557886976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312376389371392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68135331 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302137584272088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ªůqd2À- _wandbruntime‚4405  _runtime‚4405)æÖ©éÁõM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308270450570240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312458201943040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92563823 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚3649 $ _timestamp‚1.788529022392955e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4187769375744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4405.930074921 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308270432567296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76539918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22288959 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 B8DeviceStatsMonitor.on_train_bat}¢\ch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308270432567296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76539916 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344448715328512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340069628984320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4172112964166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4188679360512 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4173020939378 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856%D«'ÂúM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4405.930074921 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308270432567296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76539918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22288959 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308270432567296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76539916 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344448715328512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340069628984320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4172112964166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4188679360512 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4173020939378 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308270450570240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312458201943040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92563823 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚3649 $ _timestamp‚1.788529022392955e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4187769375744«±Øè.2Ã- _wandbruntime‚4405  _runtime‚4405v1µ]Ä× /loss/FBt4/detection_loss‚2.7461375660366483 2loss/SRqqt1/assignment_loss‚1.9570156110450625 1loss/FBt1/assignment_loss‚0.07914554189911768 /loss/FRt3/detection_loss‚1.6539974490240696 /loss/FBt1/detection_loss‚2.4994456589221956 trainer/global_step‚3649 _runtime‚4405.932605273 _step‚300 1loss/SRqqt4/detection_loss‚2.1199524477124214 &loss/total_loss‚45.726469648055264 .loss/FRt4/assignment_loss‚3.44501272462449 epoch‚3 /loss/FRt1/assignment_loss‚3.463674667903355 /loss/FRt2/assignment_loss‚3.943377438518736 0loss/FRt3/assignment_loss‚1.6634036629430708 2loss/SRqqt4/assignment_loss‚1.6302409057194989 0loss/SRqqt1/detection_loss‚3.051336844265461 /loss/FRt2/detection_loss‚2.3944619381201195 1loss/SRqqt2/detection_loss‚2.1464966614456737 1loss/SRqqt3/detection_loss‚0.9097157110143261 .loss/FBt2/detection_loss‚2.927230104804039 2loss/SRqqt2/assignment_loss‚1.6454807661254616 /loss/FRt4/detection_loss‚2.2920021799937733 /loss/FBt3/detection_loss‚0.5995659514922987 1loss/FBt3/assignment_loss‚0.02469109681893151 1loss/FBt4/assignment_loss‚0.14717520904663514 /loss/FRt1/detection_loss‚2.8530295712607248 $ _timestamp‚1.788529022397122e+09 2loss/SRqqt3/assignment_loss‚1.3313808103773705 1loss/FBt2/assignment_loss‚0.20249912894178124×^9bÅÜ 2loss/SRqqt3/assignment_loss‚1.3313808103773705 1loss/FBt2/assignment_loss‚0.20249912894178124 /loss/FBt4/detection_loss‚2.7461375660366483 2loss/SRqqt1/assignment_loss‚1.9570156110450625 1loss/FBt1/assignment_loss‚0.07914554189911768 /loss/FRt3/detection_loss‚1.6539974490240696 /loss/FBt1/detection_loss‚2.4994456589221956 trainer/global_step‚3649 _runtime‚4405.932605273 _step‚300 1loss/SRqqt4/detection_loss‚2.1199524477124214 &loss/total_loss‚45.726469648055264 .loss/FRt4/assignment_loss‚3.44501272462449 epoch‚3 /loss/FRt1/assignment_loss‚3.463674667903355 /loss/FRt2/assignment_loss‚3.943377438518736 0loss/FRt3/assignment_loss‚1.6634036629430708 2loss/SRqqt4/assignment_loss‚1.6302409057194989 0loss/SRqqt1/detection_loss‚3.051336844265461 /loss/FRt2/detection_loss‚2.3944619381201195 1loss/SRqqt2/detection_loss‚2.1464966614456737 1loss/SRqqt3/detection_loss‚0.9097157110143261 .loss/FBt2/detection_loss‚2.927230104804039 2loss/SRqqt2/assignment_loss‚1.6454807661254616 /loss/FRt4/detection_loss‚2.2920021799937733 /loss/FBt3/detection_loss‚0.5995659514922987 1loss/FBt3/assignment_loss‚0.02469109681893151 1loss/FBt4/assignment_loss‚0.14717520904663514 /loss/FRt1/detection_loss‚2.8530295712607248 $ _timestamp‚1.788529022397122e+09¬ç঒Æ‚(:‡ ‚“ëÔDZÖß gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®p‰–SÇ‚(:È ‚“ëÔÈ›œà disk.rbd0.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 cpu‚25.23748423785447 network.sent‚ 8.728296e+06 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.sda.out‚959.65380859375 disk.sdb1.in‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 54239.734375) disk./.usagePercent‚75.95275501594617 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0$ memory_percent‚95.96388339996338! disk.sda2.out‚959.65380859375 disk.sdb.in‚ 70.9609375 disk.dm-0.out‚70786.52734375 network.recv‚ 1.560907e+06 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚959.65380859375 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875% disk./.usageGB‚2716.4384155273438 disk.sda.in‚128.625 disk.md0p1.out‚ 941.73046875( proc.memory.availableMB‚ 661.27734375 proc.cpu.threads‚17# disk.nvme1n1.out‚16546.79296875 disk.md0.out‚ 941.73046875 disk.sdb.out‚959.653808593753ì’È‚(:‡ ‘“ëÔüŠ˜à gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚138.302' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.5755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾U8ÿQÉ‚(:Æ ‘“ëÔ Žçà disk.sda1.in‚0# memory_percent‚95.9620714187622 network.recv‚ 1.567011e+06 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625# disk.nvme1n1.out‚16612.04296875 proc.cpu.threads‚17 disk.dm-0.out‚71009.01953125! disk.sdb2.out‚961.12060546875 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875% disk./.usageGB‚2716.4373779296875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.404296875( proc.memory.availableMB‚ 661.57421875" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 network.sent‚ 8.838696e+06 disk.md0p1.out‚ 943.1484375) disk./.usagePercent‚75.95272600428143" disk.nvme0n1.out‚ 54396.9765625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 943.1484375 disk.sdb.out‚961.12060546875 disk.nvme1n1.in‚0 cpu‚25.23684518293287 disk.nvme0n1.in‚0! disk.sda2.out‚961.12060546875 disk.md0p1.in‚ 199.5859375 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.sda.out‚961.12060546875a¯ÿRÊ‚(:Ç  “ëÔˆ­Ÿà disk.dm-0.in‚0 disk.sda.in‚128.625 disk.sda.out‚962.86279296875 disk.sdb1.in‚0* proc.memory.percent‚18.190526962280273( proc.memory.availableMB‚ 661.55859375! disk.nvme1n1.out‚ 16677.328125 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.md0.in‚ 199.5859375" proc.memory.rssMB‚ 2980.3359375 disk.dm-0.out‚71231.64453125# disk.nvme0n1.out‚54554.31640625$ memory_percent‚95.96216678619385 proc.cpu.threads‚17! disk.sda2.out‚962.86279296875% disk./.usageGB‚2716.4373779296875 disk.sdb.out‚962.86279296875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 cpu‚25.236111992196506 network.recv‚ 1.571517e+06 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 944.8359375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚962.86279296875 disk.nvme1n1.in‚0) disk./.usagePercent‚75.95272600428143 disk.md0p1.in‚ 199.5859375 disk.sda2.in‚128.625 disk.md0.out‚ 944.8359375 network.sent‚ 8.844812e+06Ui#U”Ë‚(:‰  “ëÔò˲à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚135.517' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.87925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖŒš•’Ì‚(:‡ ¯“ëÔ®š¦à gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚148.556' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.139 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/’çTÍ‚(:É ¯“ëÔÝ›Ëà disk.sda.in‚128.625 disk.rbd2.out‚ 1.341796875 disk.md0.in‚ 199.5859375 network.sent‚ 8.848141e+06 disk.rbd2.in‚0# disk.nvme0n1.out‚54711.55859375 disk.sda2.in‚128.625! disk.sda2.out‚964.40576171875" proc.memory.rssMB‚ 2980.3359375 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚71454.16796875 disk.md0p1.out‚ 946.31640625 disk.sda1.in‚0) disk./.usagePercent‚75.95272600428143 disk.sdb.out‚964.40576171875 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚964.40576171875! disk.nvme1n1.out‚ 16742.609375 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 946.31640625* proc.memory.percent‚18.190526962280273 network.recv‚ 1.575753e+06% disk./.usageGB‚2716.4373779296875 disk.sdb1.in‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 661.46484375 cpu‚25.235428813417304 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0$ memory_percent‚95.96273899078369 disk.sda.out‚964.40576171875“.pz2Î- _wandbruntime‚4459  _runtime‚4459æ,™iÇÏÁ _step‚301 & lr-AdamW/pg1‚0.0006412457283630984 & lr-AdamW/pg2‚0.0006412457283630984 trainer/global_step‚3699 % _timestamp‚1.7885290760730233e+09 _runtime‚4459.608550654Ç5ÝÌÐÆ % _timestamp‚1.7885290760730233e+09 _runtime‚4459.608550654 _step‚301 & lr-AdamW/pg1‚0.0006412457283630984 & lr-AdamW/pg2‚0.0006412457283630984 trainer/global_step‚3699­ ^BY2Ñ- _wandbruntime‚4459  _runtime‚4459T¬§ô%ÒìO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93783894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312269785775616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4243645312000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306144341254396 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4242735340544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22570234 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316513431087616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4436230401024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527758 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93783894 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312269785775616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4243645312000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312269767772672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69025605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885290760737956e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93810238 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301917485347760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301917468308400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4459.609607079 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4226872945996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527758 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93783894 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312269785775616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4243645312000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312269767772672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69025605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885290760737956e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93810238 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301917485347760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301917468308400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4459.609607079 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4226872945996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4244766785536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46466833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885290772714133e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4243856800768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93808725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4437615195136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4437380134912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301996485902944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227990219274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316596171646976 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4243856800768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93808725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4437615195136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4437380134912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301996485902944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227990219274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316596171646976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4244766785536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46466833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885290772714133e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¯“4#2×- _wandbruntime‚4460  _runtime‚4460 I¥IYØÓ _runtime‚4460.808420138 _step‚304 2loss/SRqqt1/assignment_loss‚1.9068960547447205 0loss/SRqqt3/detection_loss‚0.760683779437095 .loss/FBt1/detection_loss‚3.039981210231781 /loss/FBt4/detection_loss‚2.6216865136073184 trainer/global_step‚3699 .loss/FRt1/detection_loss‚2.566800426929555 1loss/FBt3/assignment_loss‚0.15152849320856868 /loss/FRt4/detection_loss‚2.5328206868712293 /loss/SRqqt4/detection_loss‚2.03350186460423 2loss/SRqqt2/assignment_loss‚1.9806800809773533 0loss/SRqqt2/detection_loss‚2.247546804673744 epoch‚3 1loss/FBt4/assignment_loss‚0.37332232662307574 0loss/FRt2/assignment_loss‚3.3831104965045533 /loss/FRt4/assignment_loss‚3.133478655371555 /loss/FRt3/detection_loss‚1.3880200712726667 % _timestamp‚1.7885290772731667e+09 /loss/FRt3/assignment_loss‚1.506966296526102 %loss/total_loss‚43.71576808876014 0loss/FRt1/assignment_loss‚2.9906074772489832 2loss/FBt1/assignment_loss‚0.029041061522002566 2loss/FBt2/assignment_loss‚0.005110805373988114 1loss/SRqqt1/detection_loss‚2.2024150458542078 .loss/FRt2/detection_loss‚2.582181603746382 2loss/SRqqt3/assignment_loss‚1.0158767826249824 1loss/SRqqt4/assignment_loss‚1.559072085849519 .loss/FBt2/detection_loss‚3.013468474149704 .loss/FBt3/detection_loss‚0.690970990806818¨Ø·ØÙØ /loss/FBt4/detection_loss‚2.6216865136073184 trainer/global_step‚3699 .loss/FRt1/detection_loss‚2.566800426929555 1loss/FBt3/assignment_loss‚0.15152849320856868 /loss/FRt4/detection_loss‚2.5328206868712293 /loss/SRqqt4/detection_loss‚2.03350186460423 2loss/SRqqt2/assignment_loss‚1.9806800809773533 0loss/SRqqt2/detection_loss‚2.247546804673744 epoch‚3 1loss/FBt4/assignment_loss‚0.37332232662307574 0loss/FRt2/assignment_loss‚3.3831104965045533 /loss/FRt4/assignment_loss‚3.13347EJîE8655371555 /loss/FRt3/detection_loss‚1.3880200712726667 % _timestamp‚1.7885290772731667e+09 /loss/FRt3/assignment_loss‚1.506966296526102 %loss/total_loss‚43.71576808876014 0loss/FRt1/assignment_loss‚2.9906074772489832 2loss/FBt1/assignment_loss‚0.029041061522002566 2loss/FBt2/assignment_loss‚0.005110805373988114 1loss/SRqqt1/detection_loss‚2.2024150458542078 .loss/FRt2/detection_loss‚2.582181603746382 2loss/SRqqt3/assignment_loss‚1.0158767826249824 1loss/SRqqt4/assignment_loss‚1.559072085849519 .loss/FBt2/detection_loss‚3.013468474149704 .loss/FBt3/detection_loss‚0.690970990806818 _runtime‚4460.808420138 _step‚304 2loss/SRqqt1/assignment_loss‚1.9068960547447205 0loss/SRqqt3/detection_loss‚0.760683779437095 .loss/FBt1/detection_loss‚3.039981210231781°M™žÚ‚(:“ ¾“ëÔ“”Ýá gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â)”ÉOÛ‚(:Ä ¾“ëÔݾéâ disk.sda1.in‚0! disk.nvme0n1.out‚ 54868.796875* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17! disk.nvme1n1.out‚ 16807.859375 disk.sdb.out‚965.78271484375 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚95.96307277679443( proc.memory.availableMB‚ 661.41015625 cpu‚25.234783737018333) disk./.usagePercent‚75.95272760418942 disk.dm-0.out‚ 71676.65625 disk.md0p1.out‚ 947.64453125 disk.rbd2.in‚0 disk.nvme0n1.in‚0% disk./.usageGB‚2716.4374351501465 disk.dm-0.in‚0 disk.sda1.out‚0! disk.sdb2.out‚965.78271484375 disk.md0p1.in‚ 199.5859375 network.sent‚ 8.866655e+06 network.recv‚ 1.579654e+06 disk.sda.out‚965.78271484375 disk.sdb1.in‚0 disk.md0.out‚ 947.64453125 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda2.in‚128.625! disk.sda2.out‚965.78271484375" proc.memory.rssMB‚ 2980.3359375 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.404296875 disk.rbd2.out‚ 1.341796875 disk.md0.in‚ 199.58593752Þ¯õ”Ü‚(:‰ Í“ëÔÁÌçà gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úØà‡IÝ‚(:¾ Í“ëÔdz§â disk.rbd0.in‚0 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375) disk./.usagePercent‚75.95272600428143% disk./.usageGB‚2716.4373779296875 disk.nvme1n1.out‚ 16905.875 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0" proc.memory.rssMB‚ 2980.3359375 disk.sdb1.in‚0! disk.sdb2.out‚967.35888671875( proc.memory.availableMB‚ 661.65234375* proc.memory.percent‚18.190526962280273 disk.sdb.out‚967.35888671875" memory_percent‚95.961594581604 disk.sda.out‚967.35888671875 disk.sdb1.out‚0 disk.md0.out‚ 949.17578125 disk.sdb.in‚ 70.9609375 network.recv‚ 1.585324e+06 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.341796875 disk.sda2.in‚128.625 disk.dm-0.out‚ 71932.046875 network.sent‚ 8.963855e+06 disk.sda1.in‚0 disk.rbd2.in‚0! disk.sda2.out‚967.35888671875 disk.md0p1.out‚ 949.17578125 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 55026.171875 disk.dm-0.in‚0 cpu‚25.2341196580817pLD“Þ‚(:ˆ Ü“ëÔ”ÄÌà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.874' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.9685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜIéNß‚(:à ܓëÔ£®øã( proc.memory.availableMB‚ 661.30078125 network.sent‚ 8.97003e+06 disk.sdb1.out‚0 disk.dm-0.in‚0 disk.sdb.out‚969.0986328125 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 55183.6171875 disk.sda2.in‚128.625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚16971.12890625$ memory_percent‚95.96374034881592 disk.rbd0.out‚ 1.404296875 disk.dm-0.out‚72154.74609375 disk.sdb2.in‚ 70.9609375" proc.memory.rssMB‚ 2980.3359375 disk.sdb2.out‚969.0986328125 disk.sda.out‚969.0986328125 network.recv‚ 1.589694e+06) disk./.usagePercent‚75.95272600428143 disk.rbd2.out‚ 1.341796875 disk.sda2.out‚969.0986328125 disk.md0.in‚ 199.5859375 disk.md0p1.in‚ 199.5859375% disk./.usageGB‚2716.4373779296875 disk.md0.out‚ 950.8515625 disk.md0p1.out‚ 950.8515625* proc.memory.percent‚18.190526962280273 cpu‚25.233324446010766 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0NÐõ)“à‚(:ˆ ë“ëÔ–¨¾à gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬iÓ©Dá‚(:¹ ë“ëÔØâóá disk.nvme0n1.in‚0$ memory_percent‚95.96145153045654* proc.memory.percent‚18.190526962280273 disk.sda.in‚128.625 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 network.sent‚ 8.975149e+06 disk.sdb2.out‚ 970.5703125 disk.sda2.out‚ 970.5703125 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2980.3359375 cpu‚25.23252183813825# disk.nvme0n1.out‚55341.19921875 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 970.5703125" disk.nvme1n1.out‚ 17036.8828125 disk.md0.out‚ 952.26953125( proc.memory.availableMB‚ 661.67578125 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 970.5703125 disk.rbd0.in‚0 disk.rbd2.out‚ 1.341796875 proc.cpu.threads‚17) disk./.usagePercent‚75.95272600428143 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 network.recv‚ 1.592696e+06% disk./.usageGB‚2716.4373779296875 disk.dm-0.out‚72378.08203125 disk.md0p1.out‚ 952.26953125 disk.sda1.out‚01õ•Û2â- _wandbruntime‚4515  _runtime‚4515ºÈÕÇãÁ & lr-AdamW/pg1‚0.0006377384619977304 & lr-AdamW/pg2‚0.0006377384619977304 trainer/global_step‚3749 % _timestamp‚1.7885291320539901e+09 _runtime‚4515.589710166 _step‚305`irïÌäÆ & lr-AdamW/pg2‚0.0006377384619977304 trainer/global_step‚3749 % _timestamp‚1.7885291320539901e+09 _runtime‚4515.589710166 _step‚305 & lr-AdamW/pg1‚0.0006377384619977304±Ê4ÿÁ2å- _wandbruntime‚4515  _runtime‚4515£à¸žò'æìO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316350740066816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4298822765568 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47058018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4494713991680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353496916498432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349001949784576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚3749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320650472803840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4282750201104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316350722063872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4299732737024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95028796 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310152023251068 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4283658176108 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4298822765568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349001973724672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78535756 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78535756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316350740066816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305868365074960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47058016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69916204 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69915584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95028796 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78535758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885291320552933e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚4515.591408283 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316350722063872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4298822765568 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47058018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4494713991680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353496916498432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349001949784576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚3749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320650472803840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4282750201104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316350722063872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4299732737024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95028796 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310152023251068 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4283658176108 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4298822765568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349001973724672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78535756 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78535756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316350740066816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305868365074960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47058016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69916204 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69915584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95028796 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78535758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885291320552933e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚4515.591408283 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316350722063872 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173609543 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885291332419505e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78555918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4300854210560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 K>DeviceStatsMonitor.on_train_batch_enÝi6’Ld/inactive_split.all.freed‚69933324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4516.779703976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316432359152640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316432341149696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95079997 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚3749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95053627 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22863243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4300854210560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4299944225792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918‹iÄz'êûM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316432359152640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316432341149696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95079997 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚3749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95053627 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22863243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4300854210560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4299944225792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349092199429120 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173609543 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885291332419505e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78555918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4300854210560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69933324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4516.779703976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640³L«ŠÃ2ë- _wandbruntime‚4516  _runtime‚4516sW¾ùZìÔ /loss/FRt4/assignment_loss‚3.610979881042089 /loss/FRt1/detection_loss‚3.0871267868922305 /loss/FBt4/detection_loss‚2.5062646086399374 % _timestamp‚1.7885291332470086e+09 2loss/SRqqt1/assignment_loss‚1.8061514342984846 2loss/SRqqt3/assignment_loss‚0.8594882284690227 1loss/SRqqt2/detection_loss‚2.5385307757942766 /loss/FBt3/detection_loss‚0.6478198351235025 1loss/FBt1/assignment_loss‚0.05759484720283581 .loss/SRqqt1/detection_loss‚2.4477459192276 1loss/SRqqt4/detection_loss‚1.9465268328785896 /loss/FBt1/detection_loss‚2.7892787516117097 /loss/FBt2/detection_loss‚2.5657757737419824 1loss/SRqqt4/assignment_loss‚1.827978374097835 _runtime‚4516.782590889 0loss/FRt3/assignment_loss‚1.3450997869173686 1loss/FBt2/assignment_loss‚0.10657980024156834 epoch‚3 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚3.2500248024096856 0loss/FRt2/assignment_loss‚3.6568003975111862 .loss/FRt2/detection_loss‚2.412918863694664 %loss/total_loss‚45.23636102433301 _step‚308 2loss/SRqqt2/assignment_loss‚1.5865834348455623 .loss/FRt3/detection_loss‚1.446505758379187 .loss/FRt4/detection_loss‚2.654130896505637 1loss/FBt3/assignment_loss‚0.05768441818780209 0loss/FBt4/assignment_loss‚1.2179476341069368 1loss/SRqqt3/detection_loss‚0.8108231825133164WYx­_íÙ 2loss/SRqqt1/assignment_loss‚1.8061514342984846 2loss/SRqqt3/assignment_loss‚0.8594882284690227 1loss/SRqqt2/detection_loss‚2.5385307757942766 /loss/FBt3/detection_loss‚0.6478198351235025 1loss/FBt1/assignment_loss‚0.05759484720283581 .loss/SRqqt1/detection_loss‚2.4477459192276 1loss/SRqqt4/detection_loss‚1.9465268328785896 /loss/FBt1/detection_loss‚2.7892787516117097 /loss/FBt2/detection_loss‚2.5657757737419824 1loss/SRqqt4/assignment_loss‚1.827978374097835 _runtime‚4516.782590889 0loss/FRt3/assignment_loss‚1.3450997869173686 1loss/FBt2/assignment_loss‚0.10657980024156834 epoch‚3 trainer/global_step‚3749 0loss/FRt1/assignment_loss‚3.2500248024096856 0loss/FRt2/assignment_loss‚3.6568003975111862 .loss/FRt2/detection_loss‚2.412918863694664 %loss/total_loss‚45.23636102433301 _step‚308 2loss/SRqqt2/assignment_loss‚1.5865834348455623 .loss/FRt3/detection_loss‚1.446505758379187 .loss/FRt4/detection_loss‚2.654130896505637 1loss/FBt3/assignment_loss‚0.05768441818780209 0loss/FBt4/assignment_loss‚1.2179476341069368 1loss/SRqqt3/detection_loss‚0.8108231825133164 /loss/FRt4/assignment_loss‚3.610979881042089 /loss/FRt1/detection_loss‚3.0871267868922305 /loss/FBt4/detection_loss‚2.5062646086399374 % _timestamp‚1.7885291332470086e+09´Ç<匒î‚(:‡ ú“ëÔ®òÜá gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠNbSRï‚(:Ç ú“ëÔÈÅã disk.md0p1.out‚ 953.67578125 disk.sda1.out‚0 disk.dm-0.out‚ 72600.578125 proc.cpu.threads‚17! disk.sdb2.out‚972.02978515625 disk.sdb1.in‚0# disk.nvme0n1.out‚55498.43359375$ memory_percent‚95.96221446990967" proc.memory.rssMB‚ 2980.3359375 network.recv‚ 1.59658e+06 disk.sda.in‚128.625! disk.sda2.out‚972.02978515625 disk.dm-0.in‚0) disk./.usagePercent‚75.95271075182535% disk./.usageGB‚2716.4368324279785 disk.sda2.in‚128.625 network.sent‚ 8.990684e+06 disk.sda.out‚972.02978515625 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375* proc.memory.percent‚18.190526962280273 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.341796875 disk.md0.out‚ 953.67578125 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚972.02978515625( proc.memory.availableMB‚ 661.55078125 cpu‚25.23168691449329 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚17102.14453125ñ65…”ð‚(:‰ ‰”ëÔ¶ÞÆà gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S·[Oñ‚(:Ä ‰”ëÔíÕ–â# disk.nvme0n1.out‚55655.80078125 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚973.78271484375 disk.sda.in‚128.625 disk.nvme1n1.in‚0( disk./.usagePercent‚75.9527102185227$ disk./.usageGB‚2716.436813354492! disk.sdb2.out‚973.78271484375$ memory_percent‚95.96219062805176 disk.md0.in‚ 199.5859375 disk.md0.out‚ 955.3671875 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 cpu‚25.230882548836703 network.recv‚ 1.602378e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 955.3671875 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.rbd2.in‚0! disk.nvme1n1.out‚ 17167.453125 disk.sdb1.in‚0! disk.sda2.out‚973.78271484375 disk.dm-0.out‚72823.25390625' proc.memory.availableMB‚ 661.5546875 network.sent‚ 9.088113e+06 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda.out‚973.78271484375 disk.rbd2.out‚ 1.341796875t19/”ò‚(:‰ ˜”ëÔÑâ²à gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wéXRMó‚(: ˜”ëÔ߬¤á network.recv‚ 1.606768e+06 disk.nvme0n1.in‚0 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 956.890625 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.md0.out‚ 956.890625! disk.sdb2.out‚975.34912109375* proc.memory.percent‚18.190526962280273 cpu‚25.230028174381324 network.sent‚ 9.094228e+06 disk.sdb1.out‚0 disk.md0p1.in‚ 199.5859375' proc.memory.availableMB‚ 661.8046875 disk.sda.out‚975.34912109375$ memory_percent‚95.96066474914551 disk.sda1.in‚0 disk.nvme0n1.out‚ 55823.84375 disk.sdb.in‚ 70.9609375 disk.sdb.out‚975.34912109375# disk.nvme1n1.out‚17265.19140625" proc.memory.rssMB‚ 2980.3359375 disk.sda1.out‚0 disk.rbd2.out‚ 1.341796875! disk.sda2.out‚975.34912109375 disk.dm-0.out‚73089.03515625) disk./.usagePercent‚75.95271181843066$ disk./.usageGB‚2716.436870574951 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17]?2ô- _wandbruntime‚4573  _runtime‚4573(caŠÇõÁ & lr-AdamW/pg1‚0.0006341745448496115 & lr-AdamW/pg2‚0.0006341745448496115 trainer/global_step‚3799 % _timestamp‚1.7885291900224562e+09 _runtime‚4573.558056152 _step‚309ÝÈ îÌöÆ & lr-AdamW/pg1‚0.0006341745448496115 & lr-AdamW/pg2‚0.0006341745448496115 trainer/global_step‚3799 % _timestamp‚1.7885291900224562e+09 _runtime‚4573.558056152 _step‚309µp*¦Q2÷- _wandbruntime‚4573  _runtime‚4573 uta¹%øìO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353513235006976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175817454 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314158780233376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4339535431216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338627456212 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96300042 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚4573.559598519 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79543756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324786586545664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70806036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79543758 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96273698 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320431676355072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23144142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358066411692032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354910190592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309819244802160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324787514520064 trainer/global_step‚3799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320431694358016 ?9DeviceStatsMonitor.on_train_batch_start/segIP,9ment.all.freed‚0 _step‚310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79543756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314157855219012  ®÷'ùñO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175817454 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314158780233376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4339535431216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338627456212 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96300042 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚4573.559598519 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79543756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324786586545664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70806036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79543758 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96273698 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320431676355072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23144142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358066411692032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354910190592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309819244802160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324787514520064 trainer/global_step‚3799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320431694358016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79543756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314157855219012 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353513235006976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¶ÂíÊP2ú- _wandbruntime‚4574  _runtime‚4574…½k’û&ûõM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358157548494336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885291911884162e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79563916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79563918 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324869327091712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356031650816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320513313443840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23150535 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340652704702 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96298529 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324870255079424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23149930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚3799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79563916 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314238915101406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4554087782912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320513313443840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 _step‚311 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944*ÀÍw'üúM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70823868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96298529 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320513295440896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320513313443840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 _step‚311 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358157548494336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885291911884162e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79563916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79563918 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324869327091712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356031650816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320513313443840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23150535 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340652704702 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96298529 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324870255079424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23149930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚3799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79563916 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314238915101406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4554087782912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80551756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97518600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71696962 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48264518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328924556236288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411907587072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80551758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346Xw>-ö'ðO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80551756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178070356 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328924556236288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328923628261888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362636047201792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚3849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚4626.488842774 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611503032320 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788529242953029e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80551756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97518600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71696962 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48264518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328924556236288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411907587072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80551758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324594267735040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80571916 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318245672083714 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80571916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71714760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358114769873920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329006368807936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚3849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4413029060608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396529959810 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23437547 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412119075840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97543431 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324594267735040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80571916 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318245672083714 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80571916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71714760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358114769873920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329006368807936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚3849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4413029060608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396529959810 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23437547 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412119075840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97543431 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362535805451776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670113141760 trainer/global_step‚3899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362535829391872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328593584937472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23721611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_st»9q¦sart/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328593584937472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992Ç‚L÷'ŸñO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333061597952512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4467995012096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333061597952512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72588771 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4679.382804334 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4467085040640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23721611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328593584937472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48867768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317721004256560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81559756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322171369183628 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885292958471503e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362535805451776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670113141760 trainer/global_step‚3899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362535829391872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328593584937472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4452407214918 trainer/global_step‚3899 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4469116485632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317800004811744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367297352840192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81579918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322252429066022 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180368249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81579916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4468206500864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885292970544734e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98788333 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98814703 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362626055096320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362626031156224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333143410524160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333144338511872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4451499239706 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23726743 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0–…±'¤úM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322252429066022 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180368249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81579916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4468206500864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885292970544734e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98788333 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98814703 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362626055096320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362626031156224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333143410524160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333144338511872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4451499239706 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23726743 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328675204023296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333144338511872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23727347 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚ 4680.59118914 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328675222026240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4452407214918 trainer/global_step‚3899 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4469116485632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317800004811744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367297352840192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81579918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¿±|uƒ2¥- _wandbruntime‚4680  _runtime‚46809úX¦Õ trainer/global_step‚3899 /loss/FRt3/assignment_loss‚1.562394080766991 2loss/SRqqt1/assignment_loss‚2.1343462462723255 1loss/SRqqt2/detection_loss‚2.2294276759028433 /loss/FRt1/assignment_loss‚3.218486660056644 0loss/FRt2/assignment_loss‚3.858436044185×"@3924 1loss/SRqqt4/assignment_loss‚1.374767205160525 .loss/FRt4/detection_loss‚2.534405804024293 /loss/FBt3/detection_loss‚0.5569140101016006 /loss/FBt4/detection_loss‚3.0649891644716263 .loss/FRt4/assignment_loss‚3.28393403841899 1loss/FBt3/assignment_loss‚0.08379530952122022 /loss/FRt3/detection_loss‚1.4698780780630325 0loss/SRqqt1/detection_loss‚2.458421709537506 _step‚320 2loss/SRqqt2/assignment_loss‚1.5142838331870734 1loss/SRqqt3/detection_loss‚0.8174013736796506 epoch‚3 % _timestamp‚1.7885292970569744e+09 /loss/FRt1/detection_loss‚2.6599813646740382 /loss/FRt2/detection_loss‚2.3895824452852157 1loss/SRqqt4/detection_loss‚2.4342128857970238 .loss/FBt1/detection_loss‚3.440674066543579 /loss/FBt2/detection_loss‚2.2512113650639853 1loss/FBt2/assignment_loss‚0.34115492325507546 %loss/total_loss‚45.18925445909193 2loss/SRqqt3/assignment_loss‚1.3322183189974315 1loss/FBt1/assignment_loss‚0.04103327937175436 1loss/FBt4/assignment_loss‚0.13730457675410435 _runtime‚4680.592239375ükZ`§Ú /loss/FRt1/assignment_loss‚3.218486660056644 0loss/FRt2/assignment_loss‚3.8584360441853924 1loss/SRqqt4/assignment_loss‚1.374767205160525 .loss/FRt4/detection_loss‚2.534405804024293 /loss/FBt3/detection_loss‚0.5569140101016006 /loss/FBt4/detection_loss‚3.0649891644716263 .loss/FRt4/assignment_loss‚3.28393403841899 1loss/FBt3/assignment_loss‚0.08379530952122022 /loss/FRt3/detection_loss‚1.4698780780630325 0loss/SRqqt1/detection_loss‚2.458421709537506 _step‚320 2loss/SRqqt2/assignment_loss‚1.5142838331870734 1loss/SRqqt3/detection_loss‚0.8174013736796506 epoch‚3 % _timestamp‚1.7885292970569744e+09 /loss/FRt1/detection_loss‚2.6599813646740382 /loss/FRt2/detection_loss‚2.3895824452852157 1loss/SRqqt4/detection_loss‚2.4342128857970238 .loss/FBt1/detection_loss‚3.440674066543579 /loss/FBt2/detection_loss‚2.2512113650639853 1loss/FBt2/assignment_loss‚0.34115492325507546 %loss/total_loss‚45.18925445909193 2loss/SRqqt3/assignment_loss‚1.3322183189974315 1loss/FBt1/assignment_loss‚0.04103327937175436 1loss/FBt4/assignment_loss‚0.13730457675410435 _runtime‚4680.592239375 trainer/global_step‚3899 /loss/FRt3/assignment_loss‚1.562394080766991 2loss/SRqqt1/assignment_loss‚2.1343462462723255 1loss/SRqqt2/detection_loss‚2.2294276759028433À˜ŸaR“¨‚(:ˆ Ÿ•ëÔ¾øÿà gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°‰F©‚(:» Ÿ•ëÔíõ¹á network.recv‚ 1.647991e+06 disk.md0.out‚ 970.31640625 network.sent‚ 9.438069e+06! disk./.usageGB‚2715.564453125 disk.sda.out‚ 989.337890625 disk.nvme1n1.out‚ 17918.59375 disk.rbd2.out‚ 1.341796875 disk.sda2.in‚128.625 disk.dm-0.out‚74389.62890625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sda2.out‚ 989.337890625* proc.memory.percent‚18.190526962280273 cpu‚25.223339614266433 proc.cpu.threads‚17 disk.sdb2.out‚ 989.337890625 disk.rbd0.out‚ 1.404296875 disk.nvme0n1.in‚0# disk.nvme0n1.out‚56471.03515625) disk./.usagePercent‚75.92831866139497 disk.nvme1n1.in‚0 disk.md0p1.out‚ 970.31640625 disk.sdb1.out‚0 disk.sdb.out‚ 989.337890625 disk.sda1.in‚0# memory_percent‚95.9629774093628" proc.memory.rssMB‚ 2980.3359375 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 661.42578125 disk.dm-0.in‚0 disk.sda1.out‚0 disk.md0.in‚ 199.5859375ˆ Í-”ª‚(:‰ ®•ëÔÜÓìÞ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.995' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.74875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SJgÏL«‚(:Á ®•ëÔãˆâß network.recv‚ 1.652797e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875# disk.nvme0n1.out‚56628.27734375 disk.dm-0.in‚0 disk.dm-0.out‚74612.11328125 disk.sdb2.out‚991.0595703125 cpu‚25.222650738235206% disk./.usageGB‚2715.9401626586914 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 network.sent‚ 9.534904e+06 disk.md0.out‚ 971.953125 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17" disk.nvme1n1.out‚ 17983.8359375 disk.rbd0.out‚ 1.404296875 disk.md0p1.out‚ 971.953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.93882365720364 disk.sda1.in‚0 disk.md0.in‚ 199.5859375# memory_percent‚95.9669828414917( proc.memory.availableMB‚ 660.76953125 disk.sda2.in‚128.625 disk.sda2.out‚991.0595703125 disk.sdb.out‚991.0595703125 disk.sda.out‚991.0595703125Ñ®³‚”¬‚(:‰ ½•ëÔ‘ÒíÞ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 kÂðP­‚(:Å ½•ëÔþ¤âß! disk.sdb2.out‚992.37548828125 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0# disk.nvme0n1.out‚56785.66796875" proc.memory.rssMB‚ 2980.3359375 disk.sda.in‚128.625 disk.sda.out‚992.37548828125! disk.sda2.out‚992.37548828125* proc.memory.percent‚18.190526962280273 disk.sdb.out‚992.37548828125 disk.md0p1.in‚ 199.5859375) disk./.usagePercent‚75.93879517884156 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 18049.46875 disk.sdb1.in‚0 cpu‚25.221947472612428 disk.rbd0.in‚0 disk.nvme0n1.in‚0$ memory_percent‚95.96524238586426 disk.md0.out‚ 973.2265625 disk.dm-0.in‚0 disk.md0p1.out‚ 973.2265625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚128.625' proc.memory.availableMB‚ 661.0546875 network.recv‚ 1.657638e+06% disk./.usageGB‚2715.9391441345215 disk.md0.in‚ 199.5859375 disk.dm-0.out‚74835.13671875 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 9.541106e+06Ò-9ç2®- _wandbruntime‚4732  _runtime‚4732FjšÇ¯Á _step‚321 & lr-AdamW/pg1‚0.0006231503739572478 & lr-AdamW/pg2‚0.0006231503739572478 trainer/global_step‚3949 % _timestamp‚1.7885293489583004e+09 _runtime‚4732.493983398Â&O^Ì°Æ % _timestamp‚1.7885293489583004e+09 _runtime‚4732.493983398 _step‚321 & lr-AdamW/pg1‚0.0006231503739572478 & lr-AdamW/pg2‚0.0006231503739572478 trainer/global_step‚3949ÁJ] Ä2±- _wandbruntime‚4732  _runtime‚4732©§õ'²ïO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332674557231616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332674557231616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100008404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4523172465664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82567756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012363 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82567756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73483381 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4524082437120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49471016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367047090674176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚4732.495260954 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602506 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100034748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4507167196540 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182576160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788529348959331e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82567758 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011759 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚05Kš0ú'³ôO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4523172465664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321671883983760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100034748 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326179051180300 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332674557231616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332674557231616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100008404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4523172465664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82567756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012363 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82567756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73483381 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4524082437120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49471016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367047090674176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚4732.495260954 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602506 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100034748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4507167196540 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182576160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788529348959331e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82567758 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011759 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚323 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73501202 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332756158314496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24017508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4508284470026 _runtime‚ 4733.45465881 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.aó'Q Ällocated‚371867234824704 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4524293925888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49483083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4525203910656 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4524293925888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49483083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4525203910656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚323 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73501202 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Ãr¸YC2·- _wandbruntime‚4733  _runtime‚4733•ß¥Ì\¸Ö 1loss/FBt3/assignment_loss‚0.06770215112508092 /loss/FRt1/detection_loss‚3.9035097459952035 1loss/SRqqt3/detection_loss‚0.9438528197093142 epoch‚3 _runtime‚4733.455789185 0loss/FRt4/assignment_loss‚2.7601094160761153 1loss/FBt4/assignment_loss‚0.09037974223610945 -loss/FBt3/detection_loss‚0.72329218076988 /loss/FBt4/detection_loss‚2.3959863260388374 /loss/FRt3/detection_loss‚1.6512383171726788 _step‚324 0loss/FRt1/assignment_loss‚3.1449065605799356 1loss/FBt1/assignment_loss‚0.18621719147389135 %loss/total_loss‚42.64901674337641 2loss/SRqqt1/assignment_loss‚1.9149555584246463 /loss/FRt2/detection_loss‚2.4642320443739467 0loss/FRt3/assignment_loss‚1.3312695130469308 2loss/SRqqt3/assignment_loss‚0.9732345675014787 /loss/FRt4/detection_loss‚2.0944195045601752 1loss/SRqqt4/detection_loss‚2.1540503047014536 /loss/FBt1/detection_loss‚2.0909807284673056 .loss/FBt2/detection_loss‚2.092903508589818 0loss/FRt2/assignment_loss‚3.7742364514957774 1loss/SRqqt4/assignment_loss‚1.705923671589086 0loss/FBt2/assignment_loss‚0.1411031231366254 0loss/SRqqt2/detection_loss‚2.015970758227415 trainer/global_step‚3949 2loss/SRqqt2/assignment_loss‚1.3468918504171767 0loss/SRqqt1/detection_loss‚2.681650707667524 % _timestamp‚1.7885293499205093e+09™3{êa¹Û 0loss/FRt1/assignment_loss‚3.1449065605799356 1loss/FBt1/assignment_loss‚0.18621719147389135 %loss/total_loss‚42.64901674337641 2loss/SRqqt1/assignment_loss‚1.9149555584246463 /loss/FRt2/detection_loss‚2.4642320443739467 0loss/FRt3/assignment_loss‚1.3312695130469308 2loss/SRqqt3/assignment_loss‚0.9732345675014787 /loss/FRt4/detection_loss‚2.0944195045601752 1loss/SRqqt4/detection_loss‚2.1540503047014536 /loss/FBt1/detection_loss‚2.0909807284673056 .loss/FBt2/detection_loss‚2.092903508589818 0loss/FRt2/assignment_loss‚3.7742364514957774 1loss/SRqqt4/assignment_loss‚1.705923671589086 0loss/FBt2/assignment_loss‚0.1411031231366254 0loss/SRqqt2/detection_loss‚2.015970758227415 trainer/global_step‚3949 2loss/SRqqt2/assignment_loss‚1.3468918504171767 0loss/SRqqt1/detection_loss‚2.681650707667524 % _timestamp‚1.7885293499205093e+09 1loss/FBt3/assignment_loss‚0.06770215112508092 /loss/FRt1/detection_loss‚3.9035097459952035 1loss/SRqqt3/detection_loss‚0.9438528197093142 epoch‚3 _runtime‚4733.455789185 0loss/FRt4/assignment_loss‚2.7601094160761153 1loss/FBt4/assignment_loss‚0.09037974223610945 -loss/FBt3/detection_loss‚0.72329218076988 /loss/FBt4/detection_loss‚2.3959863260388374 /loss/FRt3/detection_loss‚1.6512383171726788 _step‚324Ĥ2ç⺂(:… Ì•ëÔ⢯ß gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.84' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.46 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S|ŒÐT»‚(:É Ì•ëÔÛÓß! disk.sda2.out‚993.72216796875 disk.sda.out‚993.72216796875 cpu‚25.221249967805537 disk.sdb1.in‚0 network.sent‚ 9.547285e+06 network.recv‚ 1.662071e+06 disk.sda2.in‚128.625! disk.sdb2.out‚993.72216796875# memory_percent‚95.9662675857544 disk.sdb.out‚993.72216796875 disk.md0p1.out‚ 974.52734375 disk.sda1.in‚0( proc.memory.availableMB‚ 660.88671875" proc.memory.rssMB‚ 2980.3359375 disk.sda.in‚128.625 disk.rbd0.out‚ 1.404296875 disk.sda1.out‚0% disk./.usageGB‚2716.4392013549805 disk.nvme0n1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sdb1.out‚0) disk./.usagePercent‚75.95277698801578# disk.nvme0n1.out‚56942.91015625 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.dm-0.out‚ 75090.1640625 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚18147.25390625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.md0.out‚ 974.52734375* proc.memory.percent‚18.190526962280273ìrã’¼‚(:‡ Û•ëÔȼÕà gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾§ž¶L½‚(:Á Û•ëÔ‘ûùà proc.cpu.threads‚17 disk.sdb2.out‚995.7861328125 disk.sda.in‚128.625# disk.nvme1n1.out‚18212.68359375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sda2.out‚995.7861328125$ memory_percent‚95.96633911132812 disk.sdb.out‚995.7861328125 disk.nvme1n1.in‚0 cpu‚25.220593387073254 disk.dm-0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 network.recv‚ 1.666996e+06 disk.sdb2.in‚ 70.9609375 disk.sda.out‚995.7861328125 disk.rbd2.in‚0# disk.nvme0n1.out‚57100.19921875 disk.sda2.in‚128.625 disk.md0.out‚ 976.52734375% disk./.usageGB‚2716.4391441345215 disk.dm-0.out‚ 75312.8828125 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 network.sent‚ 9.654275e+06) disk./.usagePercent‚75.95277538810781 disk.md0p1.out‚ 976.52734375 disk.nvme0n1.in‚0# proc.memory.availableMB‚660.875* proc.memory.percent‚18.190526962280273 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0" proc.memory.rssMB‚ 2980.3359375ᔾ‚(:‰ ê•ëÔšË‰ß gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.44425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)âƒ@¿‚(:µ ê•ëÔËÌÐá disk.sda.out‚ 997.0390625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.dm-0.in‚0 disk.sdb.out‚ 997.0390625 disk.sdb2.out‚ 997.0390625 disk.nvme1n1.in‚0 network.sent‚ 9.662011e+06 network.recv‚ 1.670809e+06 disk.nvme0n1.in‚0 disk.sda2.out‚ 997.0390625 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625" disk.nvme1n1.out‚ 18277.9609375 disk.rbd0.out‚ 1.404296875$ memory_percent‚95.96636295318604* proc.memory.percent‚18.190526962280273 cpu‚25.219895861731093 disk.md0p1.out‚ 977.72265625% disk./.usageGB‚2716.4391860961914 disk.rbd2.out‚ 1.341796875 disk.md0.out‚ 977.72265625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.95277656137367 disk.dm-0.out‚ 75535.3984375 disk.sdb2.in‚ 70.9609375 disk.nvme0n1.out‚ 57257.4375 disk.sda.in‚128.625 disk.rbd0.in‚0( proc.memory.availableMB‚ 660.87109375Öæ2À- _wandbruntime‚4784  _runtime‚4784zÜfÄÁ¾ _runtime‚4784.418398012 _step‚325 % lr-AdamW/pg1‚0.000619367396551181 % lr-AdamW/pg2‚0.000619367396551181 trainer/global_step‚3999 $ _timestamp‚1.788529400882803e+09ZaíaÉÂà % lr-AdamW/pg1‚0.000619367396551181 % lr-AdamW/pg2‚0.000619367396551181 trainer/global_step‚3999 $ _timestamp‚1.788529400882803e+09 _runtime‚4784.418398012 _step‚325ÅbvO¬2Ã- _wandbruntime‚4784  _runtime‚4784ä‡wR%ÄîO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336755511522816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24302094 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330185808162608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341334753410560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4580169862144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371558399836672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330184883148244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325622763710960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4580169862144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74376995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4784.419624888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83575758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24302727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78852940088374e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336755511522816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4787302477312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚3999 _step‚326 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83575756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101253306 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74376360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4787531259392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83575756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376345678373888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50074268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341335681384960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83575758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚:ze¢0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50074266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376345931096064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371558375896576?t‹aù'ÅóO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50074268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341335681384960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83575758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50074266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376345931096064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371558375896576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336755511522816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24302094 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330185808162608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341334753410560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4580169862144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371558399836672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330184883148244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325622763710960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4580169862144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74376995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4784.419624888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83575758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24302727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78852940088374e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336755511522816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4787302477312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚3999 _step‚326 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83575756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101253306 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74376360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4787531259392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83575756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376345678373888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570Æúyà”Æ‚(:‰ ù•ëÔÿë²à gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.387' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.59675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0× ìQÇ‚(:Æ ù•ëÔªÒ¡á disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 979.2109375 cpu‚25.21921399446083 disk.sda2.in‚128.625 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sdb1.in‚0 network.recv‚ 1.675467e+06 disk.dm-0.in‚0 disk.sda.in‚128.625' proc.memory.availableMB‚ 660.5234375* proc.memory.percent‚18.190526962280273 disk.md0.in‚ 199.5859375 disk.md0.out‚ 979.2109375 network.sent‚ 9.666015e+06% disk./.usageGB‚2716.4391860961914 disk.dm-0.out‚ 75780.9609375# disk.nvme1n1.out‚18343.35546875 disk.sdb1.out‚0# disk.nvme0n1.out‚57437.60546875! disk.sda2.out‚998.59033203125 disk.sdb.out‚998.59033203125! disk.sdb2.out‚998.59033203125 disk.md0p1.in‚ 199.5859375$ memory_percent‚95.96848487854004 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚ 70.9609375 disk.sda.out‚998.59033203125 disk.nvme1n1.in‚0 proc.cpu.threads‚17" proc.memory.rssMB‚ 2980.3359375 disk.rbd0.out‚ 1.404296875) disk./.usagePercent‚75.95277656137367 disk.sda1.in‚0ØgQU2È- _wandbruntime‚4785  _runtime‚4785ßݾ5ÿ&ÉùM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4788461594112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341418421944320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24307927 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341418421944320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4581291335680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83595918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83595916 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50086331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101304507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101278137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _runtime‚4785.410981086 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4581291335680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50086333 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371648601601024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053ñì=÷'ÊþM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325701764266144 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4581291335680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50086333 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371648601601024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4788461594112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341418421944320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24307927 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341418421944320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4581291335680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83595918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83595916 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50086331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101304507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101278137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _runtime‚4785.410981086 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845786196480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4635347315712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345472723101184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4636257287168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4618013731752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885294569538643e+09 _runtime‚4840.489516677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845786196480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4635347315712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345472723101184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4636257287168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4618013731752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885294569538643e+09 _runtime‚4840.489516677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ÊNÍw;2Ú- _wandbruntime‚4842  _runtime‚4842;víÛúM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340918084899840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334271774998374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340918084899840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4846963772928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102549409 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885294584882178e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555463660544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_poo›~9Ïl.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4619131005030 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381007099110912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4636468775936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376159910763520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340918084899840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334271774998374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340918084899840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4846963772928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102549409 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885294584882178e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555463660544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4619131005030 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381007099110912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4636468775936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376159910763520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85591756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24883278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333524506126000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4674798961864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4673890986860 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51280766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385SE 485736376832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338199322127224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85591756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24883278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333524506126000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4674798961864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4673890986860 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51280766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385485736376832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338199322127224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380671172045824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4675008260138 trainer/global_step‚4099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349692505376768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349692505376768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349691577389056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344999039191040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344999021188096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103767941 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76182456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4905932506112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85611918 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103767941 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338278531980682 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385576877880320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4692556200960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24889623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333603523720544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24889004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189379857 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380671195985920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333603540759904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360âWäÌ'ðþM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189379857 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380671195985920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333603540759904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189406229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4693466185728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4905705834496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚ 4900.12954255 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380671172045824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4675008260138 trainer/global_step‚4099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349692505376768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349692505376768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349691577389056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344999039191040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344999021188096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103767941 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76182456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4905932506112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85611918 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103767941 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338278531980682 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385576877880320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4692556200960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24889623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333603523720544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24889004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572Ïw±¸~2ñ- _wandbruntime‚4900  _runtime‚4900 ˆ^ÅlòØ .loss/FRt2/assignment_loss‚3.44465741285911 1loss/SRqqt3/assignment_loss‚1.114247404721876 .,"Ðòloss/FBt2/detection_loss‚2.188740094502767 _runtime‚4900.130557065 .loss/FRt1/detection_loss‚2.824912527674123 1loss/SRqqt1/detection_loss‚2.2032546220641387 1loss/SRqqt3/detection_loss‚0.9091983316342036 /loss/FBt3/detection_loss‚0.6630816540584482 0loss/FRt3/assignment_loss‚1.2698580208032026 1loss/FBt2/assignment_loss‚0.12208753136826545 1loss/FBt3/assignment_loss‚0.12952033584941822 1loss/FBt4/assignment_loss‚0.38244346680585295 /loss/FRt3/detection_loss‚1.2705192165854184 1loss/SRqqt4/detection_loss‚1.9853358957916498 1loss/FBt1/assignment_loss‚0.05994768908729182 &loss/total_loss‚43.250254419964165 % _timestamp‚1.7885295165952568e+09 0loss/FRt1/assignment_loss‚2.7450830873690153 /loss/FRt4/detection_loss‚2.1934559637424993 /loss/FBt4/detection_loss‚2.9936867505311966 epoch‚3 2loss/SRqqt2/assignment_loss‚1.3539099544286728 1loss/SRqqt4/assignment_loss‚1.663801606123646 /loss/FRt2/detection_loss‚2.7876860477608463 1loss/SRqqt2/detection_loss‚2.7241977590780992 trainer/global_step‚4099 _step‚336 /loss/FBt1/detection_loss‚3.0699084069993763 0loss/FRt4/assignment_loss‚3.6654805704659106 2loss/SRqqt1/assignment_loss‚1.4852400696591328Ì×ûÌcóÝ % _timestamp‚1.7885295165952568e+09 0loss/FRt1/assignment_loss‚2.7450830873690153 /loss/FRt4/detection_loss‚2.1934559637424993 /loss/FBt4/detection_loss‚2.9936867505311966 epoch‚3 2loss/SRqqt2/assignment_loss‚1.3539099544286728 1loss/SRqqt4/assignment_loss‚1.663801606123646 /loss/FRt2/detection_loss‚2.7876860477608463 1loss/SRqqt2/detection_loss‚2.7241977590780992 trainer/global_step‚4099 _step‚336 /loss/FBt1/detection_loss‚3.0699084069993763 0loss/FRt4/assignment_loss‚3.6654805704659106 2loss/SRqqt1/assignment_loss‚1.4852400696591328 .loss/FRt2/assignment_loss‚3.44465741285911 1loss/SRqqt3/assignment_loss‚1.114247404721876 .loss/FBt2/detection_loss‚2.188740094502767 _runtime‚4900.130557065 .loss/FRt1/detection_loss‚2.824912527674123 1loss/SRqqt1/detection_loss‚2.2032546220641387 1loss/SRqqt3/detection_loss‚0.9091983316342036 /loss/FBt3/detection_loss‚0.6630816540584482 0loss/FRt3/assignment_loss‚1.2698580208032026 1loss/FBt2/assignment_loss‚0.12208753136826545 1loss/FBt3/assignment_loss‚0.12952033584941822 1loss/FBt4/assignment_loss‚0.38244346680585295 /loss/FRt3/detection_loss‚1.2705192165854184 1loss/SRqqt4/detection_loss‚1.9853358957916498 1loss/FBt1/assignment_loss‚0.05994768908729182 &loss/total_loss‚43.250254419964165ÐÁ h·“ô‚(:ˆ ñ–ëÔÏõ¦à gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð"íPõ‚(:Å ñ–ëÔ«Ãá disk.rbd0.in‚0 disk.rbd0.out‚ 1.62060546875 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.dm-0.out‚ 77583.65625 disk.sdb1.in‚0 disk.sda.out‚1011.3408203125# disk.nvme1n1.out‚18931.26171875 disk.md0p1.in‚ 199.5859375 network.sent‚ 1.000829e+07 disk.nvme1n1.in‚0* proc.memory.percent‚18.190526962280273 network.recv‚ 1.711878e+06 disk.sdb1.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚58652.39453125" proc.memory.rssMB‚ 2980.3359375% disk./.usageGB‚2716.4379920959473 disk.sda1.in‚0! disk.sda2.out‚1011.3408203125 disk.sdb2.in‚ 70.9609375& proc.memory.availableMB‚ 660.390625 cpu‚25.214030469167128 disk.md0.out‚ 991.5078125 disk.sdb.out‚1011.3408203125! disk.sdb2.out‚1011.3408203125 disk.sda.in‚128.625 proc.cpu.threads‚17$ memory_percent‚95.96929550170898) disk./.usagePercent‚75.95274317662711 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 991.5078125››ZÓ”ö‚(:‰ €—ëÔ½‡¢ß gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÝŒN÷‚(:à €—ëԕéà disk.sda1.in‚0 disk.sdb2.out‚1012.873046875 disk.sda.in‚128.625 disk.rbd0.in‚0" disk.nvme0n1.out‚ 58778.4453125$ memory_percent‚95.96991539001465' proc.memory.availableMB‚ 660.2890625* proc.memory.percent‚18.190526962280273 network.sent‚ 1.0103281e+07$ disk./.usageGB‚2716.438045501709 disk.dm-0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚1012.873046875 network.recv‚ 1.717782e+06 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.md0p1.out‚ 992.9765625 disk.rbd0.out‚ 1.62060546875 disk.sdb1.in‚0 disk.md0.out‚ 992.9765625 disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625" proc.memory.rssMB‚ 2980.3359375 disk.sda.out‚1012.873046875 disk.nvme0n1.in‚0 cpu‚25.21349083290976 proc.cpu.threads‚17) disk./.usagePercent‚75.95274466987456 disk.dm-0.out‚77774.91796875 disk.sdb.out‚1012.873046875 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚18996.47265625 disk.md0p1.in‚ 199.5859375Õïqá”ø‚(:‰ —ëÔß¡Žç gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QnóJù‚(:¿ —ëÔ⾚ê network.sent‚ 1.0108614e+07) disk./.usagePercent‚75.95274306996657 disk.rbd2.out‚ 1.62109375 disk.md0.in‚ 199.5859375 disk.sdb.out‚1014.2587890625 disk.md0p1.out‚994.3125$ memory_percent‚95.96989154815674 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd2.in‚0 disk.md0.out‚994.3125 disk.dm-0.in‚0 disk.dm-0.out‚78102.64453125# disk./.usageGB‚2716.43798828125 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1014.2587890625 disk.rbd0.in‚0 disk.nvme0n1.in‚0 network.recv‚ 1.721549e+06 disk.sda2.in‚128.625! disk.sda2.out‚1014.2587890625( proc.memory.availableMB‚ 660.29296875* proc.memory.percent‚18.190526962280273 disk.sda1.out‚0# disk.nvme1n1.out‚19062.04296875 disk.sdb1.in‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 59040.6015625 disk.sdb.in‚ 70.9609375" proc.memory.rssMB‚ 2980.3359375 cpu‚25.212942078720776 proc.cpu.threads‚17! disk.sdb2.out‚1014.2587890625 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0c˜|(”ú‚(:‰ ž—ëÔ›ªêÞ gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿O4ÚMû‚(: ž—ëÔ¾Š§à disk.sdb.in‚ 70.9609375# disk.nvme0n1.out‚59168.60546875 proc.cpu.threads‚17! disk.sda.out‚1016.19384765625 disk.dm-0.out‚78328.45703125! disk.sdb.out‚1016.19384765625 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 1.62109375* proc.memory.percent‚18.190526962280273" disk.nvme1n1.out‚ 19159.8515625 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.68359375 disk.sda.in‚128.625# disk./.usageGB‚2716.43798828125 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚996.1875 disk.rbd2.in‚0 cpu‚25.212404441872085 disk.md0.in‚ 199.5859375 disk.md0.out‚996.1875 disk.rbd0.in‚0 network.recv‚ 1.725422e+06 disk.dm-0.in‚0" disk.sdb2.out‚1016.19384765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0" disk.sda2.out‚1016.19384765625$ memory_percent‚95.97060680389404) disk./.usagePercent‚75.95274306996657" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 1.011429e+07 disk.nvme0n1.in‚0 disk.sda2.in‚128.625( proc.memory.availableMB‚ 660.17578125@«2ü- _wandbruntime‚4955  _runtime‚4955c>ÁÆýÀ trainer/global_step‚4149 $ _timestamp‚1.788529571939393e+09 _runtime‚4955.474859147 _step‚337 & lr-AdamW/pg1‚0.0006077047389926151 & lr-AdamW/pg2‚0.0006077047389926151ÞÂ5ËþÅ & lr-AdamW/pg1‚0.0006077047389926151 & lr-AdamW/pg2‚0.0006077047389926151 trainer/global_step‚4149 $ _timestamp‚1.788529571939393e+09 _runtime‚4955.474859147 _step‚337ÑqeÙá2ÿ- _wandbruntime‚4955  _runtime‚4955Ý»Zö'€ðO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86599758 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747522165760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342206079109532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25174547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4963445542912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4955.476029072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348998374396416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390055701046784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729768241968 % _timestamp‚1.7885295719404194e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51884018 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353745878559232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86599756 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77059205 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77058563 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348998356393472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342205154095168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730676216972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105014356 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348998374396416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51884016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747522165760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337475402892560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191587768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191614114 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86599756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353746806533632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337475385853200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86599758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚338 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353746806533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748432137216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105014356 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0çW¥û'õO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729768241968 % _timestamp‚1.7885295719404194e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51884018 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353745878559232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86599756 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77059205 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77058563 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348998356393472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342205154095168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730676216972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105014356 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348998374396416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51884016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747522165760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337475402892560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191587768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191614114 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86599756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353746806533632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337475385853200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86599758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚338 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353746806533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748432137216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105014356 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86599758 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747522165760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342206079109532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25174547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4963445542912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4955.476029072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348998374396416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390055701046784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784Ò®xYé2‚- _wandbruntime‚4956  _runtime‚4956œ\åú‘ ƒúM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77077091 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342286213977562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4956.466790053 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353828619105280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25180368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105039213 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749553610752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748643625984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342285288962990 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51896083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86619916 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964385127424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353829547092992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748643625984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚339 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86619918 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353829547092992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390146842395648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349079975479296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105012843 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385182481208320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768±zÑ'„ÿM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77077091 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342286213977562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4956.466790053 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353828619105280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25180368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105039213 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749553610752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748643625984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342285288962990 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51896083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86619916 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964385127424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353829547092992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4748643625984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚339 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86619918 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353829547092992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390146842395648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349079975479296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105012843 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385182481208320ÓU¹3ó2…- _wandbruntime‚4956  _runtime‚4956G[˜ Z†Ô 1loss/SRqqt3/assignment_loss‚1.056879897045823 2loss/SRqqt4/assignment_loss‚1.8264573240062085 trainer/global_step‚4149 _runtime‚ 4956.46832067 /loss/FRt2/detection_loss‚2.6944177016177524 1loss/SRqqt2/detection_loss‚2.2312936261296272 _step‚340 /loss/FRt4/assignment_loss‚3.856093980231375 2loss/SRqqt1/assignment_loss‚1.8164684077103932 2loss/SRqqt2/assignment_loss‚1.7164904456585646 1loss/FBt3/assignment_loss‚0.08088697708820188 /loss/FRt1/assignment_loss‚3.017601167783141 .loss/FRt4/detection_loss‚2.304738291873122 %loss/total_loss‚48.17650099059237 0loss/FBt1/assignment_loss‚2.5165473222732544 1loss/FBt4/assignment_loss‚0.18412534990920904 .loss/FBt2/detection_loss‚2.343907193704085 0loss/FRt3/assignment_loss‚1.2619016124651983 /loss/FRt3/detection_loss‚1.6923623484774277 .loss/FBt1/detection_loss‚4.599129617214203 epoch‚3 0loss/FRt2/assignment_loss‚3.7183721488521946 0loss/FBt2/assignment_loss‚0.1382454104656445 /loss/FRt1/detection_loss‚2.8927745539695024 /loss/FBt3/detection_loss‚0.6813081961170019 /loss/FBt4/detection_loss‚2.0448863245546818 % _timestamp‚1.7885295729330535e+09 1loss/SRqqt1/detection_loss‚2.3489582123178425 1loss/SRqqt3/detection_loss‚0.9542515808193202 1loss/SRqqt4/detection_loss‚2.1984033003085997q4­_‡Ù 0loss/FBt1/assignment_loss‚2.5165473222732544 1loss/FBt4/assignment_loss‚0.18412534990920904 .loss/FBt2/detection_loss‚2.343907193704085 0loss/FRt3/assignment_loss‚1.2619016124651983 /loss/FRt3/detection_loss‚1.6923623484774277 .loss/FBt1/detection_loss‚4.599129617214203 epoch‚3 0loss/FRt2/assignment_loss‚3.7183721488521946 0loss/FBt2/assignment_loss‚0.1382454104656445 /loss/FRt1/detection_loss‚2.8927745539695024 /loss/FBt3/detection_loss‚0.6813081961170019 /loss/FBt4/detection_loss‚2.0448863245546818 % _timestamp‚1.7885295729330535e+09 1loss/SRqqt1/detection_loss‚2.3489582123178425 1loss/SRqqt3/detection_loss‚0.9542515808193202 1loss/SRqqt4/detection_loss‚2.1984033003085997 1loss/SRqqt3/assignment_loss‚1.056879897045823 2loss/SRqqt4/assignment_loss‚1.8264573240062085 trainer/global_step‚4149 _runtime‚ 4956.46832067 /loss/FRt2/detection_loss‚2.6944177016177524 1loss/SRqqt2/detection_loss‚2.2312936261296272 _step‚340 /loss/FRt4/assignment_loss‚3.856093980231375 2loss/SRqqt1/assignment_loss‚1.8164684077103932 2loss/SRqqt2/assignment_loss‚1.7164904456585646 1loss/FBt3/assignment_loss‚0.08088697708820188 /loss/FRt1/assignment_loss‚3.017601167783141 .loss/FRt4/detection_loss‚2.304738291873122 %loss/total_loss‚48.17650099059237Ô­)1Ë’ˆ‚(:‡ ­—ëÔÚ‡âß gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ø§.‰‚(:£ ­—ëÔþГã disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 997.3046875 network.sent‚ 1.0120496e+07 disk.rbd0.out‚ 1.68359375 disk.sdb2.in‚ 70.9609375 network.recv‚ 1.730198e+06) disk./.usagePercent‚75.95274306996657 disk.rbd2.in‚0$ memory_percent‚95.97015380859375# disk./.usageGB‚2716.43798828125 disk.sda1.out‚0# disk.nvme0n1.out‚59355.33203125 disk.md0.in‚ 199.5859375 disk.sdb.out‚1017.375 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.190526962280273 disk.rbd0.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625 disk.md0.out‚ 997.3046875 disk.dm-0.in‚0 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚ 78580.6015625 disk.sda.out‚1017.375# disk.nvme1n1.out‚19225.26953125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚1017.375 disk.sdb2.out‚1017.375 disk.nvme1n1.in‚0" proc.memory.availableMB‚660.25" proc.memory.rssMB‚ 2980.3359375 cpu‚25.211835933068294 disk.sdb1.out‚0äÞã6”Š‚(:‰ ¼—ëÔíÁòÞ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î §N‹‚(:à ¼—ëÔóèÚà disk.sdb1.in‚0* proc.memory.percent‚18.190526962280273 disk.sdb.in‚ 70.9609375 disk.sda.out‚1018.8193359375 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 998.69921875 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚2716.438045501709 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚19290.48046875' proc.memory.availableMB‚ 660.8046875 disk.dm-0.out‚ 78645.8125 network.sent‚ 1.0228567e+07! disk.sdb2.out‚1018.8193359375 disk.rbd2.out‚ 1.62109375 cpu‚25.21132670934936 disk.dm-0.in‚0 disk.md0p1.out‚ 998.69921875 disk.sdb1.out‚0 disk.sda1.in‚0 network.recv‚ 1.735884e+06# disk.nvme0n1.out‚59355.33203125" proc.memory.rssMB‚ 2980.3359375 disk.sda.in‚128.625$ memory_percent‚95.96676826477051 disk.sdb.out‚1018.8193359375) disk./.usagePercent‚75.95274466987456! disk.sda2.out‚1018.8193359375 disk.rbd0.in‚0ŽþÒº”Œ‚(:‰ Ë—ëÔçý¾à gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R¼÷£J‚(:¿ Ë—ëÔß„õà disk.dm-0.in‚0 disk.dm-0.out‚78953.40234375 disk.rbd0.in‚0 disk.sdb1.in‚0* proc.memory.percent‚18.190526962280273 disk.sdb.out‚1020.486328125! disk.nvme1n1.out‚ 19388.390625 disk.rbd0.out‚ 1.68359375# disk.nvme0n1.out‚59565.01171875 disk.md0.out‚ 1000.3203125 disk.md0p1.out‚ 1000.3203125 disk.sda1.out‚0) disk./.usagePercent‚75.95274306996657 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0$ memory_percent‚95.96681594848633 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sda2.out‚1020.486328125 network.recv‚ 1.73965e+06 disk.sdb2.out‚1020.486328125 disk.sda.in‚128.625 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 660.796875 network.sent‚ 1.0234459e+07 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.nvme0n1.in‚0# disk./.usageGB‚2716.43798828125 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1020.486328125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375" proc.memory.rssMB‚ 2980.3359375 cpu‚25.210792497512077IL†2Ž- _wandbruntime‚5009  _runtime‚5009½C/ŒÇÁ trainer/global_step‚4199 % _timestamp‚1.7885296261975706e+09 _runtime‚5009.733032434 _step‚341 & lr-AdamW/pg1‚0.0006037153119121266 & lr-AdamW/pg2‚0.0006037153119121266ÑØãÌÆ & lr-AdamW/pg1‚0.0006037153119121266 & lr-AdamW/pg2‚0.0006037153119121266 trainer/global_step‚4199 % _timestamp‚1.7885296261975706e+09 _runtime‚5009.733032434 _step‚341Õ‰9~¥2‘- _wandbruntime‚5009  _runtime‚5009¥%'º’ðO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25465536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106259258 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚4199 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4803609590784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5022199454208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353079328687616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357882920275456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87607756 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885296261983216e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77952804 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _step‚342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394625495846912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346212836091840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5009.733921378 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341426282619760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87607758 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341426265580400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52487268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25464918 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4804519562240 N@DeviceStatsMonitor.on_train_batch_start/a{¿k< llocation.all.allocated‚ 193867016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193840670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52487266 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346211911077476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4785645497076 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353079310684672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106232914 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4804519562240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77952184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357883848249856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87607756 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚389603516786176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106232914 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394625740180480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5021979060736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786553472080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394625495846912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346212836091840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5009.733921378 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341426282619760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87607758 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341426265580400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52487268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25464918 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4804519562240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193867016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185Ö‚¨Œz“”‚(:ˆ Ú—ëÔì÷ªà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§IÎÝP•‚(:Å Ú—ëÔ£êÔá network.recv‚ 1.74352e+06 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.dm-0.out‚79228.30078125! disk.sda.out‚1022.26123046875 disk.md0p1.in‚ 199.5859375$ memory_percent‚95.96920013427734# disk.nvme0n1.out‚59774.66015625% disk./.usageGB‚2716.4379959106445 network.sent‚ 1.0252449e+07 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1002.015625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375" disk.sda2.out‚1022.26123046875 disk.md0.out‚ 1002.015625 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1022.26123046875 disk.sda.in‚128.625 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 19453.640625 disk.sda1.out‚0 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2980.3359375 cpu‚25.21025582347326 proc.cpu.threads‚17 disk.sda2.in‚128.625% proc.memory.availableMB‚ 660.40625* proc.memory.percent‚18.190526962280273) disk./.usagePercent‚75.95274328328763 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0" disk.sdb2.out‚1022.26123046875ýµ¸2–- _wandbruntime‚5010  _runtime‚5010ž‡Zvÿ&—ùM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87627918 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394716901049856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346292970959870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚353160947773440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786762770354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚5010.784361805 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106257745 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389693766430720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52499333 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4805641035776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353160947773440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353160929770496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52499331 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341505283174944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106284115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77970604 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106257745 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87627916 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341505300214304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788529627247774e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4804731051008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712BUT „!˜þM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4805641035776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353160947773440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353160929770496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52499331 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341505283174944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106284115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77970604 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106257745 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87627916 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341505300214304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788529627247774e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4804731051008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87627918 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394716901049856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346292970959870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚353160947773440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786762770354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚5010.784361805 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106257745 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389693766430720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52499333׳82™- _wandbruntime‚5010  _runtime‚5010kÁi@\šÖ 1loss/FBt2/assignment_loss‚0.23524837982646715 .loss/FRt2/detection_loss‚2.371336384910249 0loss/FRt4/assignment_loss‚2.8196136635892532 1loss/FBt1/assignment_loss‚0.32468184375633985 /loss/FRt1/detection_loss‚3.0155197074919036 0loss/SRqqt1/detection_loss‚2.420490846489415 /loss/FBt2/detection_loss‚2.2350417137145997 /loss/FRt2/assignment_loss‚3.735056052284856 2loss/SRqqt1/assignment_loss‚1.3652358158971325 /loss/FRt3/detection_loss‚1.4352256204275524 .loss/FBt3/detection_loss‚0.635656029721837 /loss/FRt4/detection_loss‚2.2469422332942486 epoch‚3 0loss/FRt3/assignment_loss‚1.4601514216731577 1loss/SRqqt3/assignment_loss‚1.068810183335753 1loss/FBt3/assignment_loss‚0.03712363521228125 1loss/FBt4/assignment_loss‚0.20399538518309288 2loss/SRqqt4/assignment_loss‚1.0814973182148404 1loss/SRqqt4/detection_loss‚2.5047027609414525 /loss/FBt1/detection_loss‚2.5070538429113536 %loss/total_loss‚43.27622242585251 _runtime‚5010.786979956 _step‚344 1loss/SRqqt3/detection_loss‚0.9990501124411821 trainer/global_step‚4199 /loss/FBt4/detection_loss‚2.4099655584855513 % _timestamp‚1.7885296272515025e+09 1loss/SRqqt2/detection_loss‚2.4048473238945007 /loss/FRt1/assignment_loss‚4.146711074944698 1loss/SRqqt2/assignment_loss‚1.612265517210795k†ŽFa›Û /loss/FRt2/assignment_loss‚3.735056052284856 2loss/SRqqt1/assignment_loss‚1.3652358158971325 /loss/FRt3/detection_loss‚1.4352256204275524 .loss/FBt3/detection_loss‚0.635656029721837 /loss/FRt4/detection_loss‚2.2469422332942486 epoch‚3 0loss/FRt3/assignment_loss‚1.4601514216731577 1loss/SRqqt3/assignment_loss‚1.068810183335753 1loss/FBt3/assignment_loss‚0.03712363521228125 1loss/FBt4/assignment_loss‚0.20399538518309288 2loss/SRqqt4/assignment_loss‚1.0814973182148404 1loss/SRqqt4/detection_loss‚2.5047027609414525 /loss/FBt1/detection_loss‚2.5070538429113536 %loss/total_loss‚43.27622242585251 _runtime‚5010.786979956 _step‚344 1loss/SRqqt3/detection_loss‚0.9990501124411821 trainer/global_step‚4199 /loss/FBt4/detection_loss‚2.4099655584855513 % _timestamp‚1.7885296272515025e+09 1loss/SRqqt2/detection_loss‚2.4048473238945007 /loss/FRt1/assignment_loss‚4.146711074944698 1loss/SRqqt2/assignment_loss‚1.612265517210795 1loss/FBt2/assignment_loss‚0.23524837982646715 .loss/FRt2/detection_loss‚2.371336384910249 0loss/FRt4/assignment_loss‚2.8196136635892532 1loss/FBt1/assignment_loss‚0.32468184375633985 /loss/FRt1/detection_loss‚3.0155197074919036 0loss/SRqqt1/detection_loss‚2.420490846489415 /loss/FBt2/detection_loss‚2.2350417137145997Ø´6Ç0“œ‚(:ˆ é—ëÔý³Ÿà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â%B‚(:· é—ëÔ ÃÐá disk.rbd0.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 660.43359375 network.recv‚ 1.74871e+06 disk.sda.in‚128.625" disk.nvme1n1.out‚ 19518.9609375 disk.md0p1.in‚ 199.5859375 cpu‚25.20953251533075 network.sent‚ 1.0335987e+07) disk./.usagePercent‚75.95274328328763 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚ 1023.640625" proc.memory.rssMB‚ 2980.3359375 disk.dm-0.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sdb2.out‚ 1023.640625 disk.md0p1.out‚ 1003.3359375$ memory_percent‚95.96903324127197 disk.rbd0.out‚ 1.68359375 disk.sda1.out‚0 disk.sda2.out‚ 1023.640625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1003.3359375* proc.memory.percent‚18.190526962280273 disk.sda.out‚ 1023.640625 disk.nvme1n1.in‚0# disk.nvme0n1.out‚59774.72265625 disk.rbd2.in‚0 disk.dm-0.out‚79293.68359375 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17% disk./.usageGB‚2716.4379959106445 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375ŽÉY’ž‚(:‡ ø—ëÔ“Àµà gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚143.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.8625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é²Þ0JŸ‚(:¿ ø—ëÔÓ…á disk.nvme0n1.out‚ 60036.78125 network.recv‚ 1.753774e+06! disk.nvme1n1.out‚ 19584.171875 disk.rbd0.in‚0 disk.sda2.in‚128.625$ memory_percent‚95.96900939941406* proc.memory.percent‚18.190526962280273 disk.dm-0.in‚0 disk.sda.in‚128.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375) disk./.usagePercent‚75.95274488319562 disk.dm-0.out‚ 79620.953125 disk.sdb.in‚ 70.9609375 disk.sda.out‚1025.0458984375 disk.rbd2.in‚0! disk.sda2.out‚1025.0458984375 network.sent‚ 1.0435377e+07 disk.sdb.out‚1025.0458984375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0$ proc.memory.availableMB‚660.4375 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.62109375 disk.md0.out‚ 1004.6796875" proc.memory.rssMB‚ 2980.3359375 cpu‚25.208917989953715 disk.sdb1.in‚0 disk.md0p1.out‚ 1004.6796875 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0% disk./.usageGB‚2716.4380531311035! disk.sdb2.out‚1025.0458984375Àä-“ ‚(:ˆ ‡˜ëÔ«¦—à gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H®§^Y¡‚(:Î ‡˜ëÔˇá) disk./.usagePercent‚75.95274328328763 disk.sda.in‚128.625! disk.sda.out‚1026.93115234375 disk.rbd0.in‚0 disk.dm-0.in‚0* proc.memory.percent‚18.190526962280273 cpu‚25.20830135630882" disk.sdb2.out‚1026.93115234375# disk.nvme1n1.out‚19682.27734375 disk.rbd2.out‚ 1.62109375! disk.sdb.out‚1026.93115234375 disk.nvme1n1.in‚0 disk.sdb1.in‚0$ memory_percent‚95.96898555755615 disk.sdb.in‚ 70.9609375 disk.md0.out‚ 1006.51171875 network.sent‚ 1.0438668e+07% disk./.usageGB‚2716.4379959106445 disk.sdb1.out‚0 disk.dm-0.out‚ 79876.5859375" proc.memory.rssMB‚ 2980.3359375 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1006.51171875 disk.rbd2.in‚0# disk.nvme0n1.out‚60194.30859375" disk.sda2.out‚1026.93115234375 disk.rbd0.out‚ 1.68359375 disk.sda1.out‚0 disk.sda2.in‚128.625( proc.memory.availableMB‚ 660.44140625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.nvme0n1.in‚0 network.recv‚ 1.757564e+06èqÃ2¢- _wandbruntime‚5067  _runtime‚5067w{;Ç£Á & lr-AdamW/pg2‚0.0005996763244223698 trainer/global_step‚4249 % _timestamp‚1.7885296837027557e+09 _runtime‚5067.238234173 _step‚345 & lr-AdamW/pg1‚0.0005996763244223698"è†Ì¤Æ trainer/global_step‚4249 % _timestamp‚1.7885296837027557e+09 _runtime‚5067.238234173 _step‚345 & lr-AdamW/pg1‚0.0005996763244223698 & lr-AdamW/pg2‚0.0005996763244223698Ùà)Ù2¥- _wandbruntime‚5067  _runtime‚5067vÄ&ö'¦ðO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357160282978816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357160264975872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4842430727188 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4841522752184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107504160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107477816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5067.239368578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53090516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5080739440640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚4249 % _timestamp‚1.7885296837037504e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5080951445504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196093572 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107504160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78847112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399195777394176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394114802008576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53090518 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107477816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362020889966080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88615758 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350219593074148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88615756 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25756594 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88615756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350218668059784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362019961991680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357160282978816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357160264975872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196119918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399195541449216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399195777394176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394114802008576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53090518 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107477816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362020889966080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88615758 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350219593074148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88615756 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25756594 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88615756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350218668059784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362019961991680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357160282978816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357160264975872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196119918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399195541449216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394205027713024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 % _timestamp‚1.7885296850406797e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399286924503040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4843548000674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0›±ÖQ'ªÿM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4843548000674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362103630525440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350299727942178 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚347 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25762318 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399287179309056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394205051653120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394205027713024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 % _timestamp‚1.7885296850406797e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399286924503040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Û¦Çî2«- _wandbruntime‚5068  _runtime‚5068EÙ W¬Ñ 2loss/SRqqt4/assignment_loss‚1.6583458672870288 1loss/FBt3/assignment_loss‚0.14294114526396112 /loss/FRt2/detection_loss‚2.5162394178127565 /loss/FRt3/detection_loss‚1.3630586029340823 trainer/global_step‚4249 _step‚348 .loss/FRt1/assignment_loss‚3.69448314110438 2loss/FBt4/assignment_loss‚0.010942168454884419 /loss/FBt2/detection_loss‚2.8565923730532328 $loss/total_loss‚44.8814247613329 1loss/FBt2/assignment_loss‚0.22863757130101778 .loss/FBt4/detection_loss‚3.079048146804174 0loss/FRt3/assignment_loss‚0.9083824237187703 2loss/SRqqt1/assignment_loss‚2.1128539351686353 2loss/SRqqt2/assignment_loss‚1.5752211652092032 .loss/FRt1/detection_loss‚2.799513429403305 1loss/SRqqt3/detection_loss‚0.8534754711602416 /loss/FRt4/assignment_loss‚3.473827310583808 1loss/SRqqt1/detection_loss‚2.2247682059964826 .loss/FRt4/detection_loss‚2.348894600841132 1loss/SRqqt2/detection_loss‚2.0958782402244776 epoch‚3 _runtime‚ 5068.57958012 0loss/FRt2/assignment_loss‚3.7920556881211023 1loss/SRqqt3/assignment_loss‚1.121017326933465 0loss/SRqqt4/detection_loss‚2.246935939246958 .loss/FBt1/detection_loss‚3.175350947813554 /loss/FBt3/detection_loss‚0.4784157186048105 % _timestamp‚1.7885296850440307e+09 /loss/FBt1/assignment_loss‚0.124545924291438žÜ¹\­Ö 1loss/FBt3/assignment_loss‚0.14294114526396112 /loss/FRt2/detection_loss‚2.5162394178127565 /loss/FRt3/detection_loss‚1.3630586029340823 trainer/global_step‚4249 _step‚348 .loss/FRt1/assignment_loss‚3.69448314110438 2loss/FBt4/assignment_loss‚0.010942168454884419 /loss/FBt2/detection_loss‚2.8565923730532328 $loss/total_loss‚44.8814247613329 1loss/FBt2/assignment_loss‚0.22863757130101778 .loss/FBt4/detection_loss‚3.079048146804174 0loss/FRt3/assignment_loss‚0.9083824237187703 2loss/SRqqt1/assignment_loss‚2.1128539351686353 2loss/SRqqt2/assignment_loss‚1.5752211652092032 .loss/FRt1/detection_loss‚2.799513429403305 1loss/SRqqt3/detection_loss‚0.8534754711602416 /loss/FRt4/assignment_loss‚3.473827310583808 1loss/SRqqt1/detection_loss‚2.2247682059964826 .loss/FRt4/detection_loss‚2.348894600841132 1loss/SRqqt2/detection_loss‚2.0958782402244776 epoch‚3 _runtime‚ 5068.57958012 0loss/FRt2/assignment_loss‚3.7920556881211023 1loss/SRqqt3/assignment_loss‚1.121017326933465 0loss/SRqqt4/detection_loss‚2.246935939246958 .loss/FBt1/detection_loss‚3.175350947813554 /loss/FBt3/detection_loss‚0.4784157186048105 % _timestamp‚1.7885296850440307e+09 /loss/FBt1/assignment_loss‚0.124545924291438 2loss/SRqqt4/assignment_loss‚1.6583458672870288Ü4­!“®‚(:ˆ –˜ëÔ׺à gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i‚NzV¯‚(:Ë –˜ëÔ¨õ¡ã disk.sda.in‚128.625 disk.nvme1n1.in‚0" disk.sda2.out‚1028.34521484375! disk.sda.out‚1028.34521484375 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1028.34521484375# disk.nvme0n1.out‚60237.12109375 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.md0.out‚ 1007.87109375$ memory_percent‚95.96843719482422# disk.nvme1n1.out‚19747.51953125% disk./.usageGB‚2716.4379959106445 disk.sdb1.out‚0 disk.md0p1.out‚ 1007.87109375 disk.sdb1.in‚0 disk.sda1.in‚0 cpu‚25.207664684665403! disk.sdb.out‚1028.34521484375 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.in‚0% proc.memory.availableMB‚ 660.53125" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 network.recv‚ 1.760683e+06 disk.rbd0.in‚0 disk.rbd2.out‚ 1.62109375 network.sent‚ 1.0459597e+07) disk./.usagePercent‚75.95274328328763 disk.dm-0.out‚ 79972.890625 disk.sdb2.in‚ 70.9609375 disk.sda1.out‚0 disk.sda2.in‚128.625:žJJ“°‚(:ˆ ¥˜ëÔä¤Èà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÅL±‚(:Á ¥˜ëÔ“ýÿá disk.nvme0n1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 1009.3515625 network.recv‚ 1.766041e+06 disk.sdb.out‚1029.892578125 disk.sdb1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚60456.44140625 disk.sda2.in‚128.625 disk.sda2.out‚1029.892578125 disk.md0.in‚ 199.5859375 disk.rbd0.in‚0$ disk./.usageGB‚2716.353904724121 disk.sdb2.out‚1029.892578125" disk.nvme1n1.out‚ 19812.8203125* proc.memory.percent‚18.190526962280273 network.sent‚ 1.055337e+07 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.62109375 disk.md0.out‚ 1009.3515625# memory_percent‚95.9683895111084 cpu‚25.207068029172778 proc.cpu.threads‚17 disk.sda.out‚1029.892578125 disk.sdb1.in‚0' proc.memory.availableMB‚ 660.5390625 disk.dm-0.out‚80269.26171875 disk.md0p1.in‚ 199.5859375" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.95039205851755 disk.dm-0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0æAþ”²‚(:‰ ´˜ëÔë«Àà gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á6ôW³‚(:Ì ´˜ëÔ˜ëÿá disk.sda2.in‚128.625 disk.nvme1n1.in‚0 disk.md0p1.out‚ 1011.2109375 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375" disk.nvme1n1.out‚ 19867.7734375 disk.rbd2.in‚0 disk.md0.out‚ 1011.2109375 network.recv‚ 1.770535e+06 disk.nvme0n1.in‚0! disk.sdb.out‚1031.80615234375* proc.memory.percent‚18.190526962280273 network.sent‚ 1.0559625e+07 disk.dm-0.out‚80481.70703125 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 1.68359375 disk.sda1.out‚0( proc.memory.availableMB‚ 660.80078125" proc.memory.rssMB‚ 2980.3359375) disk./.usagePercent‚75.93792248236774$ disk./.usageGB‚2715.907932281494 disk.md0.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚95.96679210662842 disk.rbd2.out‚ 1.62109375# disk.nvme0n1.out‚60613.93359375" disk.sda2.out‚1031.80615234375 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1031.80615234375 disk.sda.in‚128.625! disk.sda.out‚1031.80615234375 cpu‚25.206461884299493,Ä”´‚(:‰ ØëÔâºß gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÝ¢Wµ‚(:Ì ØëÔ±¦ƒâ network.sent‚ 1.0564996e+07# disk.nvme1n1.out‚19868.67578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.62109375* proc.memory.percent‚18.190526962280273 cpu‚25.205810326071315 proc.cpu.threads‚17# disk.nvme0n1.out‚60724.19140625$ disk./.usageGB‚2715.907829284668" disk.sdb2.out‚1033.16357421875 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0" disk.sda2.out‚1033.16357421875 disk.md0.out‚ 1012.5078125" proc.memory.rssMB‚ 2980.3359375 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 80578.3671875 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0! disk.sdb.out‚1033.16357421875 disk.nvme0n1.in‚0 network.recv‚ 1.774399e+06 disk.md0p1.out‚ 1012.5078125 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚95.96903324127197( proc.memory.availableMB‚ 660.43359375) disk./.usagePercent‚75.93791960253337! disk.sda.out‚1033.16357421875 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625ÑQâ]2¶- _wandbruntime‚5124  _runtime‚5124 Ü×OÄ·¾ trainer/global_step‚4299 % _timestamp‚1.7885297412466352e+09 _runtime‚ 5124.78224146 _step‚349 % lr-AdamW/pg1‚0.000595588618242284 % lr-AdamW/pg2‚0.000595588618242284o£Fɸà _step‚349 % lr-AdamW/pg1‚0.000595588618242284 % lr-AdamW/pg2‚0.000595588618242284 trainer/global_step‚4299 % _timestamp‚1.7885297412466352e+09 _runtime‚ 5124.78224146Ý4ýoæ2¹- _wandbruntime‚5124  _runtime‚5124 ƒ eºïO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361241237270016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53693768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788529741248333e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79740804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398626111171072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5139651727872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361241219267072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349328042074160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/over¤ö¦úsize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108749062 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89623758 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403765762898944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4897400007292 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108749062 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4916694412288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚350 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403765516468224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366157003707904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89623756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108722718 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89623756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4915784440832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0?ã ú'»ôO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108749062 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4916694412288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚350 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403765516468224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366157003707904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89623756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108722718 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89623756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4915784440832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361241237270016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53693768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788529741248333e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79740804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398626111171072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5139651727872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361241219267072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349328042074160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108749062 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89623758 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403765762898944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4897400007292 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144Þs2¼- _wandbruntime‚5126  _runtime‚5126Ç—½Üþ&½øM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108747549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366239744253952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361322838352896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108773919 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89643918 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53705831 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.788529742842281e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349407059668704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚4299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26052820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366240672241664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354306484924486 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398716336875520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 _runtime‚ 5126.37892029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79759284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4916905901056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5140612123136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89643916 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚351 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 _runtime‚ 5126.37892029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79759284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4916905901056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5140612123136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89643916 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚351 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53705831 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.788529742842281e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349407059668704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚4299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26052820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small2%]Dà_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366240672241664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354306484924486 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398716336875520ß|W2¿- _wandbruntime‚5126  _runtime‚5126ÿŠ=Ò]À× 1loss/FBt2/assignment_loss‚0.18957690483269593 /loss/FRt4/detection_loss‚2.4057606297523475 1loss/SRqqt3/detection_loss‚0.9724772874439178 /loss/FBt2/detection_loss‚2.4828598548968634 % _timestamp‚1.7885297428447309e+09 _runtime‚5126.380108305 .loss/FBt4/detection_loss‚4.135451227426529 1loss/FBt4/assignment_loss‚0.14940859796479344 /loss/FRt4/assignment_loss‚2.880621481773465 1loss/FBt1/assignment_loss‚0.15034096523757196 2loss/SRqqt3/assignment_loss‚1.0602248863798978 0loss/SRqqt2/detection_loss‚2.329220677912235 2loss/SRqqt2/assignment_loss‚1.5448015433037654 /loss/FBt1/detection_loss‚2.5874352289570703 epoch‚3 trainer/global_step‚4299 2loss/SRqqt1/assignment_loss‚1.9035970715598927 1loss/SRqqt4/detection_loss‚1.9741796664163178 /loss/FRt3/detection_loss‚1.3511533057734149 /loss/FRt1/detection_loss‚3.2141102677897404 /loss/FRt2/detection_loss‚2.7377258826471524 0loss/FRt1/assignment_loss‚3.3987242736314474 0loss/FRt2/assignment_loss‚3.6932353487721197 0loss/SRqqt4/assignment_loss‚1.49836257720987 0loss/SRqqt1/detection_loss‚2.636438700887892 /loss/FBt3/detection_loss‚0.5915227314191205 _step‚352 0loss/FRt3/assignment_loss‚1.4340123169457735 1loss/FBt3/assignment_loss‚0.05934892686037604 %loss/total_loss‚45.38059035579427ÖY:ŒbÁÜ /loss/FRt4/detection_loss‚2.4057606297523475 1loss/SRqqt3/detection_loss‚0.9724772874439178 /loss/FBt2/detection_loss‚2.4828598548968634 % _timestamp‚1.7885297428447309e+09 _runtime‚5126.380108305 .loss/FBt4/detection_loss‚4.135451227426529 1loss/FBt4/assignment_loss‚0.14940859796479344 /loss/FRt4/assignment_loss‚2.880621481773465 1loss/FBt1/assignment_loss‚0.15034096523757196 2loss/SRqqt3/assignment_loss‚1.0602248863798978 0loss/SRqqt2/detection_loss‚2.329220677912235 2loss/SRqqt2/assignment_loss‚1.5448015433037654 /loss/FBt1/detection_loss‚2.5874352289570703 epoch‚3 trainer/global_step‚4299 2loss/SRqqt1/assignment_loss‚1.9035970715598927 1loss/SRqqt4/detection_loss‚1.9741796664163178 /loss/FRt3/detection_loss‚1.3511533057734149 /loss/FRt1/detection_loss‚3.2141102677897404 /loss/FRt2/detection_loss‚2.7377258826471524 0loss/FRt1/assignment_loss‚3.3987242736314474 0loss/FRt2/assignment_loss‚3.6932353487721197 0loss/SRqqt4/assignment_loss‚1.49836257720987 0loss/SRqqt1/detection_loss‚2.636438700887892 /loss/FBt3/detection_loss‚0.5915227314191205 _step‚352 0loss/FRt3/assignment_loss‚1.4340123169457735 1loss/FBt3/assignment_loss‚0.05934892686037604 %loss/total_loss‚45.38059035579427 1loss/FBt2/assignment_loss‚0.18957690483269593à\Á旓‚(:ˆ Ò˜ëÔÖÇáá gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü®èšOÂ(:Ä Ò˜ëÔÝýÿâ! disk.sdb.out‚1034.74169921875 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17! disk.sda.out‚1034.74169921875 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375( proc.memory.availableMB‚ 660.39453125 network.sent‚ 1.0581236e+07 disk.rbd0.out‚ 1.68359375 disk.dm-0.in‚0' disk./.usagePercent‚75.938690438199 disk.md0.out‚ 1014.0390625 disk.dm-0.out‚ 80884.671875* proc.memory.percent‚18.190526962280273 cpu‚25.20515282532185 disk.md0p1.out‚ 1014.0390625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2980.3359375 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 19955.765625" disk.sda2.out‚1034.74169921875" disk.sdb2.out‚1034.74169921875$ memory_percent‚95.96927165985107 network.recv‚ 1.778037e+06% disk./.usageGB‚2715.9353981018066 disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 disk.nvme0n1.out‚ 60928.90625ó™öö”Ä‚(:‰ á˜ëÔÔ³¤à gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.161' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.29025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ëa¬IÅ‚(:¾ á˜ëÔ¬êœá disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1036.564453125" proc.memory.rssMB‚ 2980.3359375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.68359375 disk.sda2.out‚1036.564453125# disk.nvme1n1.out‚19956.04296875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375( proc.memory.availableMB‚ 660.51171875 proc.cpu.threads‚17 network.sent‚ 1.0678741e+07 disk.md0p1.out‚ 1015.8046875 cpu‚25.20452723325346 network.recv‚ 1.78402e+06 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.out‚ 61060.375 disk.sda2.in‚128.625 disk.md0.out‚ 1015.8046875 disk.dm-0.out‚81013.16796875 disk.sda1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚1036.564453125 disk.sda.in‚128.625$ memory_percent‚95.96855640411377 disk.sda.out‚1036.564453125* proc.memory.percent‚18.190526962280273( disk./.usagePercent‚75.9387654205531% disk./.usageGB‚2715.9380798339844 disk.dm-0.in‚0?3F’Æ‚(:‡ ð˜ëÔ¼ð·à gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨;b?VÇ‚(:Ë ð˜ëÔ‘ßöá disk.rbd2.in‚0$ memory_percent‚95.96998691558838) disk./.usagePercent‚75.94052126623288 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.788391e+06" disk.sda2.out‚1038.18408203125 disk.rbd0.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 660.27734375" proc.memory.rssMB‚ 2980.3359375" disk.sdb2.out‚1038.18408203125 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 19958.9140625 disk.nvme0n1.in‚0 network.sent‚ 1.0684917e+07$ disk./.usageGB‚2716.000877380371! disk.sdb.out‚1038.18408203125 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.62109375 disk.dm-0.out‚81225.64453125 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚61266.73046875 cpu‚25.20397664556612! disk.sda.out‚1038.18408203125* proc.memory.percent‚18.190526962280273 disk.md0p1.out‚ 1017.3671875 disk.rbd0.out‚ 1.68359375 disk.sda2.in‚128.625 disk.md0.out‚ 1017.3671875 disk.dm-0.in‚0õ”X”È‚(:‰ ÿ˜ëÔ±šÙà gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f‡3ÒLÉ‚(:Á ÿ˜ëÔ庰á disk.md0.out‚ 1018.63671875" proc.memory.rssMB‚ 2980.3359375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 20057.15625 disk.md0p1.out‚ 1018.63671875 disk.nvme0n1.in‚0$ memory_percent‚95.96924781799316 network.sent‚ 1.0688037e+07 disk.nvme1n1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚1039.509765625* proc.memory.percent‚18.190526962280273 network.recv‚ 1.79188e+06 disk.sda.out‚1039.509765625" disk.nvme0n1.out‚ 61348.7421875 disk.sdb.out‚1039.509765625 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.68359375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 cpu‚25.203393557562958) disk./.usagePercent‚75.94225972624594 disk.sdb2.out‚1039.509765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 1.62109375 disk.dm-0.out‚ 81405.8984375' proc.memory.availableMB‚ 660.3984375% disk./.usageGB‚2716.0630531311035 disk.sda1.in‚0 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0ÐK²‡2Ê- _wandbruntime‚5182  _runtime‚5182”µ¡ÇËÁ & lr-AdamW/pg1‚0.0005914530452437102 & lr-AdamW/pg2‚0.0005914530452437102 trainer/global_step‚4349 % _timestamp‚1.7885297993839183e+09 _runtime‚5182.919368349 _step‚353N‘÷zÌÌÆ & lr-AdamW/pg1‚0.0005914530452437102 & lr-AdamW/pg2‚0.0005914530452437102 trainer/global_step‚4349 % _timestamp‚1.7885297993839183e+09 _runtime‚5182.919368349 _step‚353á+ŠØ2Í- _wandbruntime‚5182  _runtime‚5182ôÉŠZõ'ÎïO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109993964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408335151093760 $ _timestamp‚1.788529799384862e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365322173558272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚4349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4972781837312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90631758 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5197778640384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353278904762000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26337912 _step‚354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109967620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90631756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971871865856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80635568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403137396393472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971871865856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80634928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54297018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90631758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365322173558272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109993964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _runtime‚5182.920458514ÆËáJú'ÏôO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _runtime‚5182.920458514 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109993964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408335151093760 $ _timestamp‚1.788529799384862e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365322173558272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚4349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4972781837312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90631758 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5197778640384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353278904762000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26337912 _step‚354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109967620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90631756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971871865856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80635568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403137396393472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971871865856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80634928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54297018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90631758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365322173558272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109993964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0â4šk2Ð- _wandbruntime‚5184  _runtime‚5184–¢\ÑúM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109992451 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358312316892222 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408426549961216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>KÛ£ü!DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90651918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109992451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5198951803392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353357939395904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚355 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110018821 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370377713957888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365403792644096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358313241906794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4973903310848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353357922356544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972993326080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954394535678 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80653474 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365403792644096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885298005099118e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5184.047315161 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90651916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972993326080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615÷Äe'ÒÿM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200644367 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353357939395904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚355 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110018821 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370377713957888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365403792644096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358313241906794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4973903310848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353357922356544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972993326080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954394535678 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80653474 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365403792644096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885298005099118e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5184.047315161 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90651916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972993326080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109992451 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358312316892222 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408426549961216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90651918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109992451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5198951803392 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369403145852416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369403127849472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 trainer/global_step‚4399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 _runtime‚5240.741229083 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412905365005824 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369403145852416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369403127849472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 trainer/global_step‚4399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 _runtime‚5240.741229083 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412905365005824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622æ5;IÏ2ä- _wandbruntime‚5241  _runtime‚5241ô:ƒí'åúM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54912333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412996747017216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111263723 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5029080751104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5029080751104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54912331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885298583071935e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YãæÿM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407738883380224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54912331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885298583071935e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54912333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412996747017216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.aÉ'Ì¥" llocated‚ 111263723 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5029080751104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5029080751104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 Bí‚(:³ Ù™ëÔ˜¢úà disk.rbd0.out‚ 1.68359375 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 62187.8828125$ memory_percent‚95.96800804138184 disk.sda.out‚ 1048.55859375 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375' proc.memory.availableMB‚ 660.6015625 network.recv‚ 1.819139e+06 disk.sdb2.out‚ 1048.55859375 disk.md0.out‚1027.375 disk.nvme1n1.in‚0 disk.rbd2.out‚ 1.62109375 proc.cpu.threads‚17 disk.sda2.out‚ 1048.55859375$ disk./.usageGB‚2715.233154296875 disk.sda1.in‚0 disk.md0.in‚ 199.5859375" proc.memory.rssMB‚ 2980.3359375 cpu‚25.200078450188975) disk./.usagePercent‚75.91905540750126 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚1027.375 disk.dm-0.out‚ 82669.3359375 disk.sdb.out‚ 1048.55859375 network.sent‚ 1.092866e+07! disk.nvme1n1.out‚ 20481.453125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚18.190526962280273 disk.dm-0.in‚0 disk.rbd0.in‚0¼‰¦”î‚(:‰ è™ëÔìÉ×à gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.257' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.81425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒUìTï‚(:É è™ëÔáÐá disk.sdb2.in‚ 70.9609375" disk.nvme1n1.out‚ 20579.2265625 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.dm-0.out‚ 82855.2734375 disk.sda.in‚128.625 disk.md0p1.out‚ 1028.98828125 network.sent‚ 1.0932669e+07 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1050.24169921875 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375# memory_percent‚95.9702730178833% disk./.usageGB‚2715.2327613830566 disk.sdb1.out‚0 disk.sda1.in‚0" disk.sda2.out‚1050.24169921875 disk.dm-0.in‚0" disk.sdb2.out‚1050.24169921875( proc.memory.availableMB‚ 660.23046875 cpu‚25.19955704193054! disk.sda.out‚1050.24169921875* proc.memory.percent‚18.190526962280273 network.recv‚ 1.82352e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.md0.out‚ 1028.98828125) disk./.usagePercent‚75.91904442146645 disk.sda2.in‚128.625 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2980.3359375 proc.cpu.threads‚17 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0! disk.nvme0n1.out‚ 62276.046875I…šA2ð- _wandbruntime‚5294  _runtime‚5294äóûÅñ¿ % lr-AdamW/pg1‚0.000583041755975722 % lr-AdamW/pg2‚0.000583041755975722 trainer/global_step‚4449 % _timestamp‚1.7885299109772656e+09 _runtime‚5294.512812636 _step‚361énlËÊòÄ % _timestamp‚1.7885299109772656e+09 _runtime‚5294.512812636 _step‚361 % lr-AdamW/pg1‚0.000583041755975722 % lr-AdamW/pg2‚0.000583041755975722 trainer/global_step‚4449é=ò 2ó- _wandbruntime‚5294  _runtime‚5294oi’ ö'ôðO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412159942898176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚5294.514562134 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378568128856576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417474958055424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82422549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92647758 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366245695989016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361180664216400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373484082140672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315250230784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112483768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366246621003380 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5065939747620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885299109785686e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26919031 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚4449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112457424 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5084046715904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570‡Sû'õõO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112483768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205131526 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚378569056830976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92647756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5084956687360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5084956687360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5315015157248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5084046715904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412159942898176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚5294.514562134 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378568128856576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417474958055424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82422549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92647758 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366245695989016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361180664216400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373484082140672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315250230784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112483768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366246621003380 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5065939747620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885299109785686e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26919031 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚4449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112457424 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92667918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5086078160896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366326755871410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378650869402624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚363 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5086078160896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361259698850304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112508625 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412250192542720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚5295.839550813 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112482255 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5067057021106 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92667916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55515583 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361259681810944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373565719229440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ´ìT'úÿM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112482255 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5067057021106 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92667916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55515583 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361259681810944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373565719229440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417566605123072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92667918 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92667918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5086078160896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366326755871410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378650869402624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚363 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5086078160896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361259698850304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112508625 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412250192542720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚5295.839550813ë󔄸2û- _wandbruntime‚5295  _runtime‚5295*’=ËZüÔ /loss/FBt3/detection_loss‚0.6338210111587412 _runtime‚5295.840651728 0loss/FRt4/assignment_loss‚3.7622635290026665 1loss/SRqqt2/detection_loss‚1.9629480894221816 epoch‚3 .loss/FRt1/assignment_loss‚4.00897483287319 1loss/FBt3/assignment_loss‚0.07598197400037589 /loss/FRt4/detection_loss‚2.4339419856667517 0loss/FRt3/assignment_loss‚1.5075694705758775 /loss/FBt2/detection_loss‚2.4579668641090393 &loss/total_loss‚45.414944989746424 1loss/SRqqt4/detection_loss‚1.8468669774857434 .loss/FBt1/detection_loss‚4.269083023071289 2loss/SRqqt4/assignment_loss‚1.6154128367250615 .loss/FRt1/detection_loss‚2.813555371376776 .loss/FRt2/detection_loss‚2.641810729236337 /loss/FRt2/assignment_loss‚3.542001294739106 2loss/SRqqt1/assignment_loss‚1.5302678811791781 1loss/FBt1/assignment_loss‚0.29123803847469387 1loss/FBt4/assignment_loss‚0.22823799283986984 1loss/SRqqt3/detection_loss‚0.6734731834344935 _step‚364 1loss/SRqqt2/assignment_loss‚1.783789430420066 1loss/SRqqt3/assignment_loss‚0.947240247234407 0loss/SRqqt1/detection_loss‚2.421791592159787 .loss/FBt4/detection_loss‚2.124493718147278 trainer/global_step‚4449 % _timestamp‚1.7885299123053331e+09 1loss/FBt2/assignment_loss‚0.12620434705154365 /loss/FRt3/detection_loss‚1.7160105693619698ôk9è_ýÙ .loss/FRt1/assignment_loss‚4.00897483287319 1loss/FBt3/assignment_loss‚0.07598197400037589 /loss/FRt4/detection_loss‚2.4339419856667517 0loss/FRt3/assignment_loss‚1.5075694705758775 /loss/FBt2/detection_loss‚2.4579668641090393 &loss/total_loss‚45.414944989746424 1loss/SRqqt4/detection_loss‚1.8468669774857434 .loss/FBt1/detection_loss‚4.269083023071289 2loss/SRqqt4/assignment_loss‚1.6154128367250615 .loss/FRt1/detection_loss‚2.813555371376776 .loss/FRt2/detection_loss‚2.641810729236337 /loss/FRt2/assignment_loss‚3.542001294739106 2loss/SRqqt1/assignment_loss‚1.5302678811791781 1loss/FBt1/assignment_loss‚0.29123803847469387 1loss/FBt4/assignment_loss‚0.22823799283986984 1loss/SRqqt3/detection_loss‚0.6734731834344935 _step‚364 1loss/SRqqt2/assignment_loss‚1.783789430420066 1loss/SRqqt3/assignment_loss‚0.947240247234407 0loss/SRqqt1/detection_loss‚2.421791592159787 .loss/FBt4/detection_loss‚2.124493718147278 trainer/global_step‚4449 % _timestamp‚1.7885299123053331e+09 1loss/FBt2/assignment_loss‚0.12620434705154365 /loss/FRt3/detection_loss‚1.7160105693619698 /loss/FBt3/detection_loss‚0.6338210111587412 _runtime‚5295.840651728 0loss/FRt4/assignment_loss‚3.7622635290026665 1loss/SRqqt2/detection_loss‚1.9629480894221816 epoch‚3ìP­kxžþ‚(:“ †šëÔ¿ç»ß gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…j¸Nÿ‚(:à †šëÔ­Êîß disk.sda.in‚128.625 disk.sdb1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚62349.20703125 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1032.4375 disk.sdb1.out‚0" proc.memory.rssMB‚ 2980.3359375" disk.sdb2.out‚1053.81689453125" disk.sda2.out‚1053.81689453125* proc.memory.percent‚18.190526962280273 disk.dm-0.out‚ 83059.109375 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚20709.90234375 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 network.recv‚ 1.832345e+06( disk./.usagePercent‚75.9184744275828! disk.sda.out‚1053.81689453125 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1053.81689453125 disk.nvme1n1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 1032.4375 network.sent‚ 1.103424e+07$ disk./.usageGB‚2715.212375640869 disk.rbd2.out‚ 1.62109375$ memory_percent‚95.97198963165283( proc.memory.availableMB‚ 659.94921875 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.198448172845744È1x¹”€‚(:‰ •šëÔçîõÞ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ê¡NV‚(:Ë •šëÔ±¡Öß disk.md0.in‚ 199.5859375" disk.sdb2.out‚1055.23974609375 disk.rbd0.out‚ 1.68359375) disk./.usagePercent‚75.92393352693784 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1033.796875 disk.dm-0.out‚83345.80859375 disk.md0p1.out‚ 1033.796875" proc.memory.rssMB‚ 2980.3359375 disk.rbd2.out‚ 1.62109375$ memory_percent‚95.97423076629639* proc.memory.percent‚18.190526962280273 disk.nvme0n1.in‚0 disk.sda2.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚0! disk.sda.out‚1055.23974609375# disk.nvme1n1.out‚20924.45703125 disk.md0p1.in‚ 199.5859375 cpu‚25.197753310758117 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375" disk.nvme0n1.out‚ 62421.3515625 network.recv‚ 1.837131e+06 disk.sdb.in‚ 70.9609375( proc.memory.availableMB‚ 659.58203125 network.sent‚ 1.1131461e+07% disk./.usageGB‚2715.4076194763184" disk.sda2.out‚1055.23974609375! disk.sdb.out‚1055.23974609375 disk.sda.in‚128.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0oùLᔂ‚(:‰ ¤šëÔº ëÞ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨@„_Mƒ‚(: ¤šëÔÌÒæß network.recv‚ 1.841538e+06) disk./.usagePercent‚75.92405309339435 disk.sda2.in‚128.625 disk.sdb.out‚1056.732421875 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1056.732421875 disk.md0p1.in‚ 199.5859375 cpu‚25.197152451758367 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0$ disk./.usageGB‚2715.411895751953 disk.sda2.out‚1056.732421875 disk.md0.out‚ 1035.23046875$ memory_percent‚95.97563743591309 disk.nvme0n1.in‚0 disk.sda.out‚1056.732421875# disk.nvme0n1.out‚62425.73828125 disk.sda.in‚128.625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚20989.69921875 disk.rbd0.out‚ 1.68359375' proc.memory.availableMB‚ 659.3515625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0* proc.memory.percent‚18.190526962280273 disk.rbd2.out‚ 1.62109375 disk.dm-0.in‚0 network.sent‚ 1.1137662e+07 disk.dm-0.out‚ 83415.4375 disk.md0p1.out‚ 1035.23046875 disk.rbd0.in‚0 disk.sdb1.in‚0" proc.memory.rssMB‚ 2980.3359375Ë% Å2„- _wandbruntime‚5346  _runtime‚5346Jð(ºÇ…Á trainer/global_step‚4499 % _timestamp‚1.7885299632210786e+09 _runtime‚5346.756581064 _step‚365 & lr-AdamW/pg1‚0.0005787677926063838 & lr-AdamW/pg2‚0.0005787677926063838Q£ï‚Ì†Æ & lr-AdamW/pg1‚0.0005787677926063838 & lr-AdamW/pg2‚0.0005787677926063838 trainer/global_step‚4499 % _timestamp‚1.7885299632210786e+09 _runtime‚5346.756581064 _step‚365í¿R“2‡- _wandbruntime‚5346  _runtime‚5346R5ôBȈðO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113702326 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422045196703744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382705170572800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416671252060672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113728670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113702326 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚4499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5373944643072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382706098547200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93655758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93655756 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422044939787264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27208510 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H>DeviceSN¸Úð.tatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113702326 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚4499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5373944643072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382706098547200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93655758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93655756 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422044939787264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27208510 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113728670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0î]ÀŸ2Š- _wandbruntime‚5348  _runtime‚5348÷Æ^Û'‹úM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5141255601152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5122934276214 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377646655517696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4499 % _timestamp‚1.7885299645112255e+09 _step‚367 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5375106930176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93675916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113727157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5141255601152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93675918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207429445 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377646673520640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93675916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113753527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207403073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382787911118848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93675918 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5122026301002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760ADq‚ŒÿM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83333761 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5122026301002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5141255601152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5122934276214 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377646655517696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4499 % _timestamp‚1.7885299645112255e+09 _step‚367 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 aODeviceStatsMonitor.on_train_batch_end/inactive_sׄíƃ plit_bytes.small_pool.allocated‚ 5375106930176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93675916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113727157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5141255601152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93675918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207429445 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377646673520640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93675916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113753527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207403073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382787911118848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93675918 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5176786282832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114947228 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426614957322240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381645990723072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374260134967996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386843140263424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421182537283072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197131537408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386842212289024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27498528Å2ä­¡&ôO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5176786282832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114947228 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426614957322240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381645990723072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374260134967996 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386843140263424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421182537283072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197131537408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386842212289024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27498528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56710016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374259209953632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84208544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27499159 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5196221565952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114973572 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5196221565952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5177694257836 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421182513342976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5432420039168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381646008726016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114947228 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small;׈‘Y_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024òõ f2ž- _wandbruntime‚5402  _runtime‚5402á…<îÿ&ŸùM _runtime‚ 5402.40038554 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5177903556110 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚371 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5178811531322 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374339344821454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386925880822784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885300188635914e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381727609808896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421272739047424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369161441265344 trainer/global_step‚4549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381727627811840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114972059 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27504321 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94683916 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84227017 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615©S$6' þM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374339344821454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386925880822784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885300188635914e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381727609808896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421272739047424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369161441265344 trainer/global_step‚4549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381727627811840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114972059 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27504321 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94683916 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84227017 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚ 5402.40038554 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5177903556110 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚371 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5178811531322 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896óÖÄ,¯2¡- _wandbruntime‚5402  _runtime‚5402ŸtºZ¢Ô 1loss/FBt3/assignment_loss‚0.03295680212457245 /loss/FRt1/detection_loss‚3.1405383944511414 .loss/FRt3/detection_loss‚1.410950579577022 /loss/FRt4/detection_loss‚2.0461622455290387 0loss/SRqqt1/detection_loss‚2.333103237407548 /loss/FRt3/assignment_loss‚1.186136803929768 0loss/FRt4/assignment_loss‚3.3883407894445927 .loss/FBt4/detection_loss‚2.142989867925644 2loss/SRqqt3/assignment_loss‚1.0302040386287605 1loss/FBt1/assignment_loss‚0.17061992178622115 1loss/FBt2/assignment_loss‚0.25173676725725613 1loss/SRqqt3/detection_loss‚0.8679283695185885 _step‚372 /loss/FRt2/assignment_loss‚3.445896887010144 /loss/SRqqt2/detection_loss‚2.27831865974835 trainer/global_step‚4549 0loss/FBt4/assignment_loss‚0.1512788667179848 .loss/FRt2/detection_loss‚2.534907880112495 2loss/SRqqt1/assignment_loss‚1.8292002316032137 2loss/SRqqt2/assignment_loss‚1.6791519869651113 %loss/total_loss‚41.11718489443735 epoch‚3 _runtime‚5402.402167028 0loss/FRt1/assignment_loss‚2.7928038196904317 1loss/SRqqt4/detection_loss‚2.3765006279572845 /loss/FBt1/detection_loss‚1.8464782346378674 % _timestamp‚1.7885300188668551e+09 2loss/SRqqt4/assignment_loss‚1.6201834697276354 /loss/FBt2/detection_loss‚1.8029953986406326 /loss/FBt3/detection_loss‚0.7578010140460426$­R‹_£Ù /loss/FRt2/assignment_loss‚3.445896887010144 /loss/SRqqt2/detection_loss‚2.27831865974835 trainer/global_step‚4549 0loss/FBt4/assignment_loss‚0.1512788667179848 .loss/FRt2/detection_loss‚2.534907880112495 2loss/SRqqt1/assignment_loss‚1.8292002316032137 2loss/SRqqt2/assignment_loss‚1.6791519869651113 %loss/total_loss‚41.11718489443735 epoch‚3 _runtime‚5402.402167028 0loss/FRt1/assignment_loss‚2.7928038196904317 1loss/SRqqt4/detection_loss‚2.3765006279572845 /loss/FBt1/detection_loss‚1.8464782346378674 % _timestamp‚1.7885300188668551e+09 2loss/SRqqt4/assignment_loss‚1.6201834697276354 /loss/FBt2/detection_loss‚1.8029953986406326 /loss/FBt3/detection_loss‚0.7578010140460426 1loss/FBt3/assignment_loss‚0.03295680212457245 /loss/FRt1/detection_loss‚3.1405383944511414 .loss/FRt3/detection_loss‚1.410950579577022 /loss/FRt4/detection_loss‚2.0461622455290387 0loss/SRqqt1/detection_loss‚2.333103237407548 /loss/FRt3/assignment_loss‚1.186136803929768 0loss/FRt4/assignment_loss‚3.3883407894445927 .loss/FBt4/detection_loss‚2.142989867925644 2loss/SRqqt3/assignment_loss‚1.0302040386287605 1loss/FBt1/assignment_loss‚0.17061992178622115 1loss/FBt2/assignment_loss‚0.25173676725725613 1loss/SRqqt3/detection_loss‚0.8679283695185885 _step‚372ô ð‘¤‚(:† ïšëÔÁúà gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02°bO¥‚(:Ä ïšëÔ«¾Ïá" disk.nvme0n1.out‚ 62793.5078125 disk.sda2.in‚128.625$ memory_percent‚95.97172737121582 cpu‚25.19450808668444 disk.sdb.out‚1064.392578125 disk.sda1.in‚0' proc.memory.availableMB‚ 659.9921875" proc.memory.rssMB‚ 2980.3359375* proc.memory.percent‚18.190526962280273 disk.nvme0n1.in‚0) disk./.usagePercent‚75.93453579715181 disk.sdb2.out‚1064.392578125 disk.md0p1.in‚ 199.5859375 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚ 1042.60546875 disk.sdb1.out‚0 disk.dm-0.out‚84175.24609375 disk.rbd0.in‚0 disk.sda2.out‚1064.392578125 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚21381.73828125 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.862618e+06 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1042.60546875 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 network.sent‚ 1.1267126e+07$ disk./.usageGB‚2715.786808013916 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda.out‚1064.392578125¢ ´ó”¦‚(:‰ þšëÔäÆ‘ß gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¿´JE§‚(:º þšëÔÇžÈà disk.dm-0.out‚ 84397.71875 disk.sda.in‚128.625 disk.md0p1.out‚ 1044.3671875 disk.sdb1.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.94851600641806 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.nvme1n1.out‚ 21446.96875" proc.memory.availableMB‚659.75 cpu‚25.194029325301784 proc.cpu.threads‚17" disk.sdb2.out‚1066.21728515625! disk.sda.out‚1066.21728515625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1044.3671875 network.recv‚ 1.86777e+06$ disk./.usageGB‚2716.286808013916 disk.nvme1n1.in‚0 disk.rbd0.out‚ 1.68359375 disk.sda1.in‚0 disk.rbd2.out‚ 1.62109375! disk.sdb.out‚1066.21728515625* proc.memory.percent‚18.190526962280273 disk.nvme0n1.out‚62950.75 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.in‚0" disk.sda2.out‚1066.21728515625 disk.dm-0.in‚0" proc.memory.rssMB‚ 2980.3359375 network.sent‚ 1.1374484e+07 disk.sdb.in‚ 70.9609375$ memory_percent‚95.97320556640625­ùy’¨‚(:‡ ›ëÔø¸à gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qæWT©‚(:É ›ëÔ¹¿ýá disk.nvme0n1.in‚0 disk.dm-0.in‚0' proc.memory.availableMB‚ 659.9921875* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 21512.3359375 cpu‚25.193575021436725 disk.md0.in‚ 199.5859375 disk.rbd0.out‚ 1.68359375 disk.md0p1.out‚ 1045.734375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375$ memory_percent‚95.97172737121582" proc.memory.rssMB‚ 2980.3359375 network.recv‚ 1.872324e+06! disk.sda.out‚1067.65087890625 disk.sda2.in‚128.625" disk.sda2.out‚1067.65087890625 disk.dm-0.out‚84672.82421875! disk.sdb.out‚1067.65087890625" disk.sdb2.out‚1067.65087890625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚63160.48828125 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 1045.734375 network.sent‚ 1.1382767e+07) disk./.usagePercent‚75.94851600641806$ disk./.usageGB‚2716.286808013916L‰æï2ª- _wandbruntime‚5457  _runtime‚54574EtRÇ«Á trainer/global_step‚4599 % _timestamp‚1.7885300737340999e+09 _runtime‚5457.269736664 _step‚373 & lr-AdamW/pg1‚0.0005700876816492022 & lr-AdamW/pg2‚0.0005700876816492022ÐqoëÌ¬Æ & lr-AdamW/pg1‚0.0005700876816492022 & lr-AdamW/pg2‚0.0005700876816492022 trainer/global_step‚4599 % _timestamp‚1.7885300737340999e+09 _runtime‚5457.269736664 _step‚373õ·HŒõ2­- _wandbruntime‚5457  _runtime‚5457ŒUIÚs®ïO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚390980181979648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5252308990976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390979254005248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373033303398000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95671758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5253218962432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116192130 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390980181979648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95671758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5491015706624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚5457.272293575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚378266891950304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5252308990976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5252308990976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚390980181979648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5252308990976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390979254005248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373033303398000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95671758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5253218962432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116192130 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390980181979648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95671758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5491015706624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚5457.272293575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚378266891950304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116216961 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95691918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385808582103040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚375 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116243331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5234688786430 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5253430451200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5233780811218 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57325331 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116243331 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27795219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95691916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378347026818334 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425784048209920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5492192599040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373112338031904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95691916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85120552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378346101803762 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116216961 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385808564100096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425784024269824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885300748474412e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712—Y‰”0²þM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5233780811218 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57325331 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116243331 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27795219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95691916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378347026818334 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425784048209920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5492192599040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373112338031904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95691916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85120552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378346101803762 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116216961 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385808564100096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425784024269824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885300748474412e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27794580 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431275988100096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116216961 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95691918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385808582103040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚375 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116243331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5234688786430 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5253430451200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615÷ŒzƵ2³- _wandbruntime‚5458  _runtime‚5458õ´ÔÞU´Ï 1loss/SRqqt3/detection_loss‚1.0868910109918368 %loss/total_loss‚42.71672624052606 1loss/SRqqt3/assignment_loss‚1.034133219210382 /loss/FBt1/detection_loss‚2.5589332468807697 _runtime‚5458.385127665 epoch‚3 % _timestamp‚1.7885300748498192e+09 2loss/SRqqt2/assignment_loss‚1.4061661936887881 2loss/SRqqt4/assignment_loss‚1.2745169626016701 /loss/FBt2/assignment_loss‚0.167986714399035 0loss/FBt4/assignment_loss‚0.2725061374968781 0loss/SRqqt2/detection_loss‚2.012839672768988 /loss/FRt1/assignment_loss‚4.081347291668256 /loss/FRt1/detection_loss‚3.1016746858755746 _step‚376 .loss/FRt2/detection_loss‚2.507693287391658 /loss/FRt2/assignment_loss‚3.664143951733907 0loss/FBt1/assignment_loss‚0.3854002992451342 .loss/FRt4/detection_loss‚2.429530434511803 1loss/SRqqt1/detection_loss‚2.3651595245236936 1loss/FBt3/assignment_loss‚0.08412026704767855 0loss/SRqqt4/detection_loss‚2.554833397269249 /loss/FBt2/detection_loss‚1.7361340451808203 /loss/FBt3/detection_loss‚0.5334005651454772 /loss/FBt4/detection_loss‚1.8426003115517753 trainer/global_step‚4599 /loss/FRt3/assignment_loss‚1.355449301501115 /loss/FRt4/assignment_loss‚2.903743036695429 .loss/FRt3/detection_loss‚1.418660430651572 2loss/SRqqt1/assignment_loss‚1.9388622524945631(ÅË{ZµÔ 1loss/SRqqt3/assignment_loss‚1.034133219210382 /loss/FBt1/detection_loss‚2.5589332468807697 _runtime‚5458.385127665 epoch‚3 % _timestamp‚1.7885300748498192e+09 2loss/SRqqt2/assignment_loss‚1.4061661936887881 2loss/SRqqt4/assignment_loss‚1.2745169626016701 /loss/FBt2/assignment_loss‚0.167986714399035 0loss/FBt4/assignment_loss‚0.2725061374968781 0loss/SRqqt2/detection_loss‚2.012839672768988 /loss/FRt1/assignment_loss‚4.081347291668256 /loss/FRt1/detection_loss‚3.1016746858755746 _step‚376 .loss/FRt2/detection_loss‚2.507693287391658 /loss/FRt2/assignment_loss‚3.664143951733907 0loss/FBt1/assignment_loss‚0.3854002992451342 .loss/FRt4/detection_loss‚2.429530434511803 1loss/SRqqt1/detection_loss‚2.3651595245236936 1loss/FBt3/assignment_loss‚0.08412026704767855 0loss/SRqqt4/detection_loss‚2.554833397269249 /loss/FBt2/detection_loss‚1.7361340451808203 /loss/FBt3/detection_loss‚0.5334005651454772 /loss/FBt4/detection_loss‚1.8426003115517753 trainer/global_step‚4599 /loss/FRt3/assignment_loss‚1.355449301501115 /loss/FRt4/assignment_loss‚2.903743036695429 .loss/FRt3/detection_loss‚1.418660430651572 2loss/SRqqt1/assignment_loss‚1.9388622524945631 1loss/SRqqt3/detection_loss‚1.0868910109918368 %loss/total_loss‚42.71672624052606øÎÙ숔¶‚(:‰ œ›ëÔ×þÞ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q†•ïM·‚(: œ›ëÔó”¬á disk.sda1.out‚0 disk.dm-0.in‚0" disk.sdb2.out‚1069.08251953125 disk.md0p1.out‚ 1047.109375" proc.memory.availableMB‚659.75 disk.sdb1.out‚0 disk.sda1.in‚0" disk.sda2.out‚1069.08251953125 disk.dm-0.out‚84927.82421875 disk.md0p1.in‚ 199.5859375 cpu‚25.193142673157737 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0* proc.memory.percent‚18.190526962280273 network.sent‚ 1.138671e+07! disk.sdb.out‚1069.08251953125! disk.sda.out‚1069.08251953125 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1047.109375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625$ memory_percent‚95.97320556640625 disk.nvme0n1.in‚0# disk.nvme1n1.out‚21610.09765625 network.recv‚ 1.87686e+06) disk./.usagePercent‚75.94851760632605 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0" disk.nvme0n1.out‚ 63317.7265625" proc.memory.rssMB‚ 2980.3359375$ disk./.usageGB‚2716.286865234375 disk.sdb1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625>ã}Ï’¸‚(:‡ «›ëÔ à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.4825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ýH¹‚(:½ «›ëÔ¿îÔà disk.sda1.out‚0# disk.nvme0n1.out‚63475.04296875 disk.sda2.in‚128.625 network.recv‚ 1.881859e+06 disk.md0.out‚ 1049.0859375 proc.cpu.threads‚17 network.sent‚ 1.1496038e+07 disk.nvme1n1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚18.190526962280273 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.62109375' proc.memory.availableMB‚ 660.1171875" proc.memory.rssMB‚ 2980.3359375 cpu‚25.19266776806215 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1071.12109375 disk.sda2.out‚ 1071.12109375# memory_percent‚95.9709644317627 disk.dm-0.out‚ 85150.6328125 disk.sda.in‚128.625 disk.sda.out‚ 1071.12109375 disk.md0p1.out‚ 1049.0859375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 1071.12109375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0) disk./.usagePercent‚75.94851685970232$ disk./.usageGB‚2716.286838531494# disk.nvme1n1.out‚21675.58984375 disk.md0.in‚ 199.5859375Í.– ”º‚(:‰ º›ëÔš¶£à gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FA“¢Q»‚(:Æ º›ëÔËö“á! disk.sda.out‚1072.70654296875 cpu‚25.19218427048772 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.97365856170654 disk.md0.out‚ 1050.60546875 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1072.70654296875 disk.nvme1n1.out‚ 21740.84375 disk.nvme1n1.in‚0 network.recv‚ 1.88586e+06) disk./.usagePercent‚75.94851675304179 disk.md0.in‚ 199.5859375! disk.sdb.out‚1072.70654296875* proc.memory.percent‚18.190526962280273$ disk./.usageGB‚2716.286834716797 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375" proc.memory.rssMB‚ 2980.3359375 disk.rbd0.in‚0" disk.sda2.out‚1072.70654296875 proc.cpu.threads‚17 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0! disk.nvme0n1.out‚ 63555.046875 disk.sda2.in‚128.625 disk.dm-0.out‚ 85295.890625 disk.md0p1.out‚ 1050.60546875( proc.memory.availableMB‚ 659.67578125 network.sent‚ 1.1501747e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 disk.nvme0n1.in‚0 disk.dm-0.in‚0˜à 2“¼‚(:ˆ É›ëÔ‹Œßà gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚149.442' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.3605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ª3)R½‚(:Ç É›ëÔ¨ëà disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.dm-0.in‚0! disk.sdb.out‚1073.98193359375" disk.sdb2.out‚1073.98193359375( proc.memory.availableMB‚ 659.67578125 proc.cpu.threads‚17 network.sent‚ 1.1505751e+07 disk.rbd2.out‚ 1.62109375! disk.nvme0n1.out‚ 63684.796875 disk.dm-0.out‚ 85490.890625 cpu‚25.191711165952185 disk.sda.in‚128.625 disk.nvme1n1.in‚0" disk.sda2.out‚1073.98193359375) disk./.usagePercent‚75.94851675304179 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0! disk.sda.out‚1073.98193359375 disk.md0p1.out‚ 1051.83984375 disk.md0.out‚ 1051.83984375 disk.sdb2.in‚ 70.9609375$ memory_percent‚95.97365856170654" proc.memory.rssMB‚ 2980.3359375 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375$ disk./.usageGB‚2716.286834716797 disk.nvme1n1.out‚ 21806.09375 disk.rbd0.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚18.190526962280273 network.recv‚ 1.89067e+06 disk.sdb1.out‚0f52¾- _wandbruntime‚5512  _runtime‚5512½E1¥Æ¿À & lr-AdamW/pg2‚0.0005656833429834722 trainer/global_step‚4649 $ _timestamp‚1.788530128495866e+09 _runtime‚5512.031367349 _step‚377 & lr-AdamW/pg1‚0.0005656833429834722ØÎfËÀÅ _runtime‚5512.031367349 _step‚377 & lr-AdamW/pg1‚0.0005656833429834722 & lr-AdamW/pg2‚0.0005656833429834722 trainer/global_step‚4649 $ _timestamp‚1.788530128495866e+09ùr½eÚ2Á- _wandbruntime‚5512  _runtime‚5512Y’õõ'ÂïO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5288540793048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5309306387456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430205107727872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376984200164560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚378 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5309306387456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389807899305472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435754804113408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788530128496645e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389807899305472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435754804113408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788530128496645e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5309306387456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430205107727872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_óŽ~oœtrain_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376984200164560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640ú>/®2Ä- _wandbruntime‚5513  _runtime‚5513Ãî«ÿ&ÅùM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214188151 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389889536394240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57928581 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395199036267520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5290566041538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117488233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57928583 $ _timestamp‚1.788530129786972e+09 _step‚379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5310427860992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5309517876224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86014665 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28086082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430295333432320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5289658066326 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395199964255232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28085443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389889518391296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117488233 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389889518391296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5310427860992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570Ñη<'ÆþM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389889536394240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5550868110848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117488233 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389889518391296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5310427860992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214188151 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389889536394240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57928581 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395199036267520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5290566041538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117488233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57928583 $ _timestamp‚1.788530129786972e+09 _step‚379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5310427860992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5309517876224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86014665 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28086082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430295333432320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5289658066326 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395199964255232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28085443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389889518391296 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚5567.622080764 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345326023160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86888902 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚&’¨=Á440324842576384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399254265412096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216396036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386279480900556 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118681934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 _step‚382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784+zú™û'×õO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86888902 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440324842576384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399254265412096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216396036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386279480900556 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118681934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 _step‚382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28369749 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386280405914920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚5567.622080764 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345326023160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0þ½zý2Ø- _wandbruntime‚5568  _runtime‚5568Twè('ÙúM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393970472682496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118733135 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚383 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚4699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118733135 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434806594714624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97707916 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393970472682496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386359615768378 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5345535321434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440415990585344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399337005971456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚393970490685440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386360540782950 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381014097486304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97707916 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58531831 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5366515286016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5609395870720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399336077983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5365605301248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885301852879276e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118706765 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393970490685440 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28374899 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58531831 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5366515286016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5609395870720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399336077983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5365605301248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885301852879276e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118706765 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393970490685440 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28374899 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚383 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚4699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118733135 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434806594714624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97707916 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393970472682496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144ÿÉ š·2Û- _wandbruntime‚5568  _runtime‚5568Mö¯qXÜÒ %loss/total_loss‚40.32901051112213 1loss/SRqqt2/assignment_loss‚1.377683503297614 1loss/SRqqt4/assignment_loss‚1.642831670462477 0loss/FBt4/assignment_loss‚0.1371015745102745 /loss/FRt4/detection_loss‚2.2677043884478767 0loss/FRt2/assignment_loss‚2.6840371780925327 /loss/FRt3/detection_loss‚1.3193001041395798 /loss/FBt4/detection_loss‚2.6590953204366894 trainer/global_step‚4699 .loss/FRt2/detection_loss‚2.346381313933278 .loss/FRt4/assignment_loss‚3.24888461211632 2loss/SRqqt1/assignment_loss‚1.4133091506858666 1loss/FBt2/assignment_loss‚0.11845657564109995 0loss/SRqqt4/detection_loss‚1.921679962654503 .loss/FBt2/detection_loss‚2.418025036652883 _step‚384 0loss/FRt1/assignment_loss‚2.7330055899090238 /loss/FBt1/detection_loss‚2.7139419967477973 $ _timestamp‚1.788530185290348e+09 1loss/SRqqt3/assignment_loss‚1.008817388362178 1loss/FBt3/assignment_loss‚0.07372327791370299 1loss/SRqqt2/detection_loss‚1.8715327732703264 /loss/FBt3/detection_loss‚0.5237650806196483 0loss/FRt3/assignment_loss‚1.2859310905138652 1loss/FBt1/assignment_loss‚0.12740993944250725 .loss/FRt1/detection_loss‚2.684660988825339 1loss/SRqqt1/detection_loss‚2.8214862681925297 1loss/SRqqt3/detection_loss‚0.9302457262542141 epoch‚3 _runtime‚5568.825631386ÈjªI]Ý× %loss/total_loss‚40.32901051112213 1loss/SRqqt2/assignment_loss‚1.377683503297614 1loss/SRqqt4/assignment_loss‚1.642831670462477 0loss/FBt4/assignment_loss‚0.1371015745102745 /loss/FRt4/detection_loss‚2.2677043884478767 0loss/FRt2/assignment_loss‚2.6840371780925327 /loss/FRt3/detection_loss‚1.3193001041395798 /loss/FBt4/detection_loss‚2.6590953204366894 trainer/global_step‚4699 .loss/FRt2/detection_loss‚2.346381313933278 .loss/FRt4/assignment_loss‚3.24888461211632 2loss/SRqqt1/assignment_loss‚1.4133091506858666 1loss/FBt2/assignment_loss‚0.11845657564109995 0loss/SRqqt4/detection_loss‚1.921679962654503 .loss/FBt2/detection_loss‚2.418025036652883 _step‚384 0loss/FRt1/assignment_loss‚2.7330055899090238 /loss/FBt1/detection_loss‚2.7139419967477973 $ _timestamp‚1.788530185290348e+09 1loss/SRqqt3/assignment_loss‚1.008817388362178 1loss/FBt3/assignment_loss‚0.07372327791370299 1loss/SRqqt2/detection_loss‚1.8715327732703264 /loss/FBt3/detection_loss‚0.5237650806196483 0loss/FRt3/assignment_loss‚1.2859310905138652 1loss/FBt1/assignment_loss‚0.12740993944250725 .loss/FRt1/detection_loss‚2.684660988825339 1loss/SRqqt1/detection_loss‚2.8214862681925297 1loss/SRqqt3/detection_loss‚0.9302457262542141 epoch‚3 _runtime‚5568.825631386€’Q ž”Þ‚(:‰ ”œëÔ­Èîá gpu.0.gpu‚34.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ rOTß‚(:É ”œëÔö¹­â" disk.nvme1n1.out‚ 22165.2265625 disk.md0p1.out‚ 1059.328125 network.sent‚ 1.1636829e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda.out‚1081.76611328125 disk.rbd0.out‚ 1.68359375 disk.sda.in‚128.625 disk.nvme1n1.in‚0) disk./.usagePercent‚75.94851323324423$ disk./.usageGB‚2716.286708831787 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 1.62109375# memory_percent‚95.9730863571167* proc.memory.percent‚18.190526962280273 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375 network.recv‚ 1.912915e+06 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1081.76611328125# disk.nvme0n1.out‚64523.65234375 disk.dm-0.out‚86688.87890625( proc.memory.availableMB‚ 659.76953125 cpu‚25.189443389725948 disk.rbd2.in‚0" disk.sda2.out‚1081.76611328125! disk.sdb.out‚1081.76611328125 disk.rbd0.in‚0 disk.md0.out‚ 1059.328125" proc.memory.rssMB‚ 2980.3359375‚|€”à‚(:‰ £œëÔŽÚ™à gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pwJNá‚(:à £œëÔÝÞ¨à network.sent‚ 1.1744043e+07 disk.rbd2.out‚ 1.62109375# disk.nvme0n1.out‚64523.79296875* proc.memory.percent‚18.190526962280273 disk.sdb1.in‚0 disk.sda.out‚1083.162109375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1060.6640625( proc.memory.availableMB‚ 659.51953125 disk.rbd0.out‚ 1.68359375 disk.sda1.out‚0 disk.sdb2.out‚1083.162109375$ memory_percent‚95.97461223602295 cpu‚25.188957022889877) disk./.usagePercent‚75.94851163333625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625# disk.nvme1n1.out‚22295.66015625 network.recv‚ 1.918165e+06 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.sdb.out‚1083.162109375 disk.sdb2.in‚ 70.9609375 disk.sda2.out‚1083.162109375 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1060.6640625 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2980.3359375$ disk./.usageGB‚2716.286651611328 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚ 86819.453125 disk.sda.in‚128.6252? ’â‚(:‡ ²œëÔûÝ®à gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.95' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V}Pã‚(:Å ²œëÔ²çà disk.rbd0.in‚0 disk.md0.in‚ 199.5859375# proc.memory.rssMB‚ 2980.33984375 network.recv‚ 1.921727e+06 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb.out‚1085.18603515625 cpu‚25.188449138820264 disk.sdb1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚22467.05859375* proc.memory.percent‚18.190550804138184) disk./.usagePercent‚75.94142169444505 disk.rbd0.out‚ 1.68359375 disk.nvme1n1.in‚0% disk./.usageGB‚2716.0330810546875 disk.sda2.in‚128.625 disk.dm-0.in‚0" disk.sdb2.out‚1085.18603515625 disk.sda.in‚128.625! disk.sda.out‚1085.18603515625 disk.md0p1.in‚ 199.5859375 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.dm-0.out‚87200.49609375 disk.md0p1.out‚ 1062.6171875$ memory_percent‚95.97625732421875 disk.sdb1.out‚0 disk.nvme0n1.out‚ 64733.4375" disk.sda2.out‚1085.18603515625 disk.md0.out‚ 1062.6171875 disk.sdb2.in‚ 70.9609375" proc.memory.availableMB‚659.25 network.sent‚ 1.1751761e+07 disk.rbd2.in‚0Œ`é2ä- _wandbruntime‚5624  _runtime‚5624˜ÇåÁ & lr-AdamW/pg1‚0.0005567506883743367 & lr-AdamW/pg2‚0.0005567506883743367 trainer/global_step‚4749 % _timestamp‚1.7885302405912023e+09 _runtime‚5624.126757249 _step‚385/*®’ÌæÆ % _timestamp‚1.7885302405912023e+09 _runtime‚5624.126757249 _step‚385 & lr-AdamW/pg1‚0.0005567506883743367 & lr-AdamW/pg2‚0.0005567506883743367 trainer/global_step‚4749ø£2ç- _wandbruntime‚5624  _runtime‚5624Ö¥dóõ'èïO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218648938 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28660061 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390287162897228 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397969807887872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 $ _timestamp‚1.788530240592789e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397969807887872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5421481237504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390286237882864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚4749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119926836 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5420571266048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59123016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚386 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59123016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚386 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 $ _timestamp‚1.788530240592789e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397969807887872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5421481237504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390286237882864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚4749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119926836 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5420571266048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98715918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5401412576542 % _timestamp‚1.7885302417685153e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5625.304490726 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87801660 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384964960174144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119978037 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444985780637696 _step‚387 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98715916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ƒîÇ'îÿM trainer/global_step‚4749 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59135083 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5422602711040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390366372750686 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98715916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398051426973696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28666577 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98715918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5401412576542 % _timestamp‚1.7885302417685153e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5625.304490726 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87801660 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384964960174144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119978037 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444985780637696 _step‚387 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121198082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402050780182016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28951095 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449464206585856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚4799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5725493815808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 GADeviceStatsMonitor.on_train_ba6Ú’!tch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394293919879536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59726268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5681.795151972 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220875494 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5477568662528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5476658691072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640Ú²aFú'ýôO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5457080533376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59726266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88677363 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388836822306800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121198082 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443738963395072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99703756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402050762179072 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121198082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402050780182016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28951095 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449464206585856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚4799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5725493815808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394293919879536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59726268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5681.795151972 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220875494 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5477568662528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5476658691072 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394373129732994 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394374054747566 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402132399267840 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚391 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407611089403904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402132399267840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59738331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121196569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59738333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885302992203512e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚4799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28956892 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99723918 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449555851931136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449555597125120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5726662831616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5458197806862 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99723918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443829189099520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28956259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5726431965696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388915856940704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5457289831650 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402132381264896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 û1€ÿM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402132381264896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407610161416192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99723916 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394373129732994 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394374054747566 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402132399267840 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚391 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407611089403904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402132399267840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59738331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121196569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59738333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885302992203512e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚4799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28956892 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99723918 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449555851931136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449555597125120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5726662831616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5458197806862 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99723918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443829189099520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28956259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5726431965696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388915856940704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5457289831650‡!l¸2- _wandbruntime‚5682  _runtime‚5682» Æ‚[‚Õ /loss/FBt3/detection_loss‚0.5469938999897724 _runtime‚5682.758100616 /loss/FBt2/detection_loss‚2.0358686993519464 &loss/total_loss‚42.798215768389866 2loss/SRqqt2/assignment_loss‚1.6259487247055706 1loss/FBt2/assignment_loss‚0.26009231200440297 /loss/FRt4/detection_loss‚2.3816837025567508 0loss/SRqqt4/detection_loss‚2.436001362623992 .loss/FBt1/detection_loss‚2.434480900565783 0loss/FRt4/assignment_loss‚2.7660927161192284 2loss/SRqqt3/assignment_loss‚1.2710086258346254 /loss/FRt3/detection_loss‚1.1682699666766638 0loss/SRqqt1/detection_loss‚2.302704833447933 trainer/global_step‚4799 2loss/FBt3/assignment_loss‚0.001984656232590254 1loss/SRqqt3/detection_loss‚0.7712346265052862 epoch‚3 1loss/SRqqt4/assignment_loss‚1.425320017117041 0loss/FBt1/assignment_loss‚0.1360577985356978 % _timestamp‚1.7885302992228231e+09 .loss/FBt4/detection_loss‚3.205366611480713 /loss/FRt1/assignment_loss‚2.984972516695658 2loss/SRqqt1/assignment_loss‚1.7437396412715316 0loss/FBt4/assignment_loss‚1.2160768845642451 _step‚392 /loss/FRt2/assignment_loss‚3.494986169040203 0loss/FRt3/assignment_loss‚1.4864809238508847 /loss/FRt1/detection_loss‚3.0705884397029877 1loss/SRqqt2/detection_loss‚1.8286630536119144 .loss/FRt2/detection_loss‚2.203598685904442j}ÀÁ`ƒÚ 1loss/SRqqt4/assignment_loss‚1.425320017117041 0loss/FBt1/assignment_loss‚0.1360577985356978 % _timestamp‚1.7885302992228231e+09 .loss/FBt4/detection_loss‚3.205366611480713 /loss/FRt1/assignment_loss‚2.984972516695658 2loss/SRqqt1/assignment_loss‚1.7437396412715316 0loss/FBt4/assignment_loss‚1.2160768845642451 _step‚392 /loss/FRt2/assignment_loss‚3.494986169040203 0loss/FRt3/assignment_loss‚1.4864809238508847 /loss/FRt1/detection_loss‚3.0705884397029877 1loss/SRqqt2/detection_loss‚1.8286630536119144 .loss/FRt2/detection_loss‚2.203598685904442 /loss/FBt3/detection_loss‚0.5469938999897724 _runtime‚5682.758100616 /loss/FBt2/detection_loss‚2.0358686993519464 &loss/total_loss‚42.798215768389866 2loss/SRqqt2/assignment_loss‚1.6259487247055706 1loss/FBt2/assignment_loss‚0.26009231200440297 /loss/FRt4/detection_loss‚2.3816837025567508 0loss/SRqqt4/detection_loss‚2.436001362623992 .loss/FBt1/detection_loss‚2.434480900565783 0loss/FRt4/assignment_loss‚2.7660927161192284 2loss/SRqqt3/assignment_loss‚1.2710086258346254 /loss/FRt3/detection_loss‚1.1682699666766638 0loss/SRqqt1/detection_loss‚2.302704833447933 trainer/global_step‚4799 2loss/FBt3/assignment_loss‚0.001984656232590254 1loss/SRqqt3/detection_loss‚0.7712346265052862 epoch‚3ˆEñZð’„‚(:‡ ýœëÔíõ›á gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gmª²V…‚(:Ë ýœëÔ¶øüá disk.dm-0.in‚0 network.recv‚ 1.944303e+06 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.rbd2.out‚ 1.62109375 cpu‚25.18578742848002 network.sent‚ 1.1960069e+07 disk.rbd2.in‚0! disk.sda2.out‚1093.5341796875$ memory_percent‚95.97527980804443( proc.memory.availableMB‚ 659.41015625# proc.memory.rssMB‚ 2980.33984375 disk.sda.out‚1093.5341796875 disk.sdb.out‚1093.5341796875) disk./.usagePercent‚75.94577120428444 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1093.5341796875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.nvme0n1.in‚0 disk.sda.in‚128.625 disk.nvme1n1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚128.625* proc.memory.percent‚18.190550804138184% disk./.usageGB‚2716.1886405944824" disk.nvme1n1.out‚ 22655.5859375# disk.nvme0n1.out‚65607.50390625 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1070.67578125 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1070.67578125 disk.dm-0.out‚88263.08984375Eù„n”†‚(:‰ ŒëÔß gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éJ.[‡‚(:Ð ŒëÔß”Ñâ disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚65637.38671875$ memory_percent‚95.97728252410889* proc.memory.percent‚18.190550804138184 cpu‚25.185294483033875% disk./.usageGB‚2716.1881141662598 disk.sdb1.out‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 659.08203125 network.recv‚ 1.949722e+06) disk./.usagePercent‚75.94575648513101 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.62109375" disk.sda2.out‚1094.90380859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2980.33984375! disk.sda.out‚1094.90380859375# disk.nvme1n1.out‚22688.37109375 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 disk.rbd0.out‚ 1.68359375 proc.cpu.threads‚17" disk.sdb2.out‚1094.90380859375 disk.nvme1n1.in‚0 disk.md0p1.out‚ 1072.00390625 network.sent‚ 1.2070287e+07 disk.sdb1.in‚0 disk.md0.out‚ 1072.00390625 disk.dm-0.out‚ 88324.7578125! disk.sdb.out‚1094.90380859375 disk.sda.in‚128.625!Oô”ˆ‚(:‰ ›ëÔÜË›à gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê?Ø>P‰‚(:Å ›ëÔÔˆðà) disk./.usagePercent‚75.95274658976415 disk.dm-0.out‚ 88653.9453125 disk.md0p1.out‚ 1073.51171875& proc.memory.availableMB‚ 659.265625 disk.sda1.out‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚22819.09765625 disk.rbd0.in‚0* proc.memory.percent‚18.190550804138184 network.sent‚ 1.2075601e+07 network.recv‚ 1.953688e+06 disk.sdb1.out‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚65834.84765625 disk.md0.out‚ 1073.51171875 disk.sdb2.out‚1096.462890625$ memory_percent‚95.97616195678711 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.md0.in‚ 199.5859375 disk.sdb.out‚1096.462890625# proc.memory.rssMB‚ 2980.33984375 disk.sda2.in‚128.625 disk.dm-0.in‚0 disk.md0p1.in‚ 199.5859375% disk./.usageGB‚2716.4381141662598 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.out‚1096.462890625 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda.out‚1096.462890625 cpu‚25.18482637111837 proc.cpu.threads‚17ɼÚ\’Š‚(:‡ ªëÔôâ¢à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îA/V‹‚(:Ë ªëÔÃÑòà) disk./.usagePercent‚75.95274658976415" disk.sda2.out‚1098.12255859375! disk.sdb.out‚1098.12255859375! disk.sda.out‚1098.12255859375 disk.nvme1n1.out‚ 22884.34375" disk.nvme0n1.out‚ 65992.0859375 disk.nvme1n1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚95.97618579864502 disk.nvme0n1.in‚0 disk.dm-0.out‚ 88876.4296875 proc.cpu.threads‚17 disk.sdb1.in‚0% disk./.usageGB‚2716.4381141662598 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1075.10546875 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 cpu‚25.18894986569985 network.recv‚ 1.957099e+06 disk.md0.out‚ 1075.10546875 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1098.12255859375 disk.sda1.in‚0# proc.memory.rssMB‚ 2980.33984375* proc.memory.percent‚18.190550804138184 network.sent‚ 1.2081181e+07 disk.md0.in‚ 199.5859375 disk.dm-0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 659.26171875'Or¥“Œ‚(:ˆ ¹ëÔ…§Åà gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²µëK‚(:À ¹ëÔ®Ùýà" disk.nvme0n1.out‚ 66149.4453125" disk.sda2.out‚1099.89892578125 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375" disk.sdb2.out‚1099.89892578125 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1076.8125 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚2716.4381141662598! disk.sdb.out‚1099.89892578125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.out‚ 22949.6875 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.md0.out‚ 1076.8125 disk.dm-0.out‚ 89099.1328125& proc.memory.availableMB‚ 659.265625 disk.sdb.in‚ 70.9609375! disk.sda.out‚1099.89892578125) disk./.usagePercent‚75.95274658976415 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 2980.33984375 network.recv‚ 1.961537e+06 disk.sda.in‚128.625 disk.rbd2.in‚0 cpu‚25.19963834592903 proc.cpu.threads‚17 network.sent‚ 1.2085224e+07 disk.rbd2.out‚ 1.62109375 disk.dm-0.in‚0$ memory_percent‚95.97616195678711* proc.memory.percent‚18.190550804138184 disk.nvme1n1.in‚0 disk.sdb1.in‚0æQÿ“Ž‚(:ˆ ÈëÔò›Æà gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w\1X‚(:Í ÈëÔ«ƒçá disk.sda1.in‚0 network.sent‚ 1.2091393e+07% disk./.usageGB‚2716.4381141662598 disk.dm-0.in‚0 disk.sda.in‚128.625# disk.nvme1n1.out‚23014.92578125 network.recv‚ 1.965748e+06! disk.sda.out‚1101.43017578125 disk.rbd0.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 1078.29296875 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 1.62109375( proc.memory.availableMB‚ 659.15234375 proc.cpu.threads‚17 disk.dm-0.out‚ 89321.609375 disk.md0p1.out‚ 1078.29296875 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb.out‚1101.43017578125 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.68359375 disk.md0.in‚ 199.5859375 cpu‚25.21082656687026" disk.sdb2.out‚1101.43017578125 disk.nvme0n1.in‚0" disk.sda2.out‚1101.43017578125# proc.memory.rssMB‚ 2980.33984375) disk./.usagePercent‚75.95274658976415 disk.rbd2.in‚0# memory_percent‚95.9768533706665* proc.memory.percent‚18.190550804138184 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚66306.68359375Åe)®“‚(:ˆ ×ëÔÙé¦à gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ŽïT‘‚(:É ×ëÔ’š½á disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0% disk./.usageGB‚2716.4381141662598 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.969805e+06" disk.sdb2.out‚1103.00732421875 disk.nvme0n1.in‚0# disk.nvme0n1.out‚66464.10546875 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 89544.453125# proc.memory.rssMB‚ 2980.33984375) disk./.usagePercent‚75.95274658976415 disk.sda.in‚128.625 disk.md0p1.out‚ 1079.8046875! disk.sda.out‚1103.00732421875 disk.sda1.out‚0" disk.sda2.out‚1103.00732421875 disk.md0.out‚ 1079.8046875! disk.sdb.out‚1103.00732421875$ memory_percent‚95.97606658935547% proc.memory.availableMB‚ 659.28125* proc.memory.percent‚18.190550804138184 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚23080.34765625 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 cpu‚25.221105167700255 network.sent‚ 1.209894e+07¦Àn2’- _wandbruntime‚5783  _runtime‚57830& a*“ÛT _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 _runtime‚5783.336205734 _step‚393 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90068712 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚60641209 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5543087774720 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885303998002305e+09 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90068932 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5792903752192 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122529080 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚449834424595456 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚449834400655360 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 101208000 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29427503 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413271857533440 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 trainer/global_step‚4835 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 101208002 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 122529080 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5542404735488 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 192344576 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 122549791 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101208002 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚413271857533440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚455627496752128 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5522365327590 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5521684260142 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚394231369498304 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122549791 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5543087774720 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 168404480 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101208000 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223737080 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5542404735488 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185øÏÞe% ”àT WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413271857533440 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 trainer/global_step‚4835 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 101208002 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 122529080 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5542404735488 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 192344576 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 122549791 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101208002 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚413271857533440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 cODeviceStatsMonitor.on_validation_batch_start/inactive_sÌžZAplit_bytes.all.allocated‚455627496752128 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5522365327590 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5521684260142 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚394231369498304 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122549791 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5543087774720 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 168404480 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101208000 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223737080 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5542404735488 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 _runtime‚5783.336205734 _step‚393 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90068712 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚60641209 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5543087774720 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885303998002305e+09 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90068932 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5792903752192 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122529080 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚449834424595456 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚449834400655360 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 101208000 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29427503 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0‰ÄTŠ 2•- _wandbruntime‚5784  _runtime‚5784º(‡j)–äR dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚449890668209152 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 _runtime‚ 5784.49727456 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _step‚394 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚455684061812224 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚60650773 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 trainer/global_step‚4835 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚29432931 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚394281778564128 % _timestamp‚1.7885304009612405e+09 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚407783731707904 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚413326608291328 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚399804649141962 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 101223478 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5522853538474 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5543577625600 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223761370 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60650771 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 168404480 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚407783731707904 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223782083 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90083923 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 122537894 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚4077837137049602v+ o)—éR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5543577625600 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚394281778564128 % _timestamp‚1.7885304009612405e+09 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚407783731707904 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚413326608291328 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚399804649141962 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 101223478 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5522853538474 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5543577625600 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223761370 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60650771 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 168404480 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚407783731707904 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223782083 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90083923 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 122537894 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚407783713704960 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚449890668209152 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 _runtime‚ 5784.49727456 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _step‚394 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚455684061812224 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚60650773 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 trainer/global_step‚4835 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚29432931 B&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1537792831659317 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962748885154724 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18857352435588837 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993098378181458 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.37914225459098816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2763386070728302 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24885372817516327 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0055158245377242565 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18936915695667267 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/ev?_ÀMùent_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41853317618370056 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9755746722221375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5100612044334412 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007286648731678724 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.34090909361839294 particle/accuracy_2_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5052679181098938 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4748897850513458 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/f_score‚0.42730259895324707 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2924995720386505 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32309505343437195 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20641084015369415 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4474544823169708 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4042319655418396 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1711111217737198 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46699419617652893 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3632037341594696 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333432674408 +jet/accuracy_2_of_2‚0.10400119423866272 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6824390292167664 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301887214183807 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.36438480019569397 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4876176118850708 +jet/accuracy_2_of_5‚0.40844854712486267 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_2_of_7‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19619466364383698 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2011563926935196 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4852164387702942 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17708133161067963 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18779945373535156 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010673235170543194 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022399630397558212 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2020440399646759 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8224474787712097 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2626262307167053 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5438172817230225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4707951247692108 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.537869393825531 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5545138120651245 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4536789357662201 (particle/accuracy‚0.8580521941184998 jet/accuracy_5_of_6‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23055556416511536 +jet/accuracy_2_of_6‚0.49358975887298584 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9856238961219788 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5373322367668152 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42988502979278564 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32685184478759766 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902342557907104 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2330024689435959 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4830748736858368 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.2666666805744171 particle/accuracy_6_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48749619722366333 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5310250520706177 particle/accuracy_2_of_2‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020364467054605484 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21423786878585815 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919083118438721 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984030008316 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4710519015789032 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.548214316368103 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7459497451782227 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01994902268052101 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3259672522544861 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 jet/accuracy_6_of_6‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.44290122389793396 particle/accuracy_3_of_4‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19722366333007812 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023085733875632286 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880220293998718 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.18653270602226257 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19173391163349152 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3741742968559265 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2568965256214142 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08289338648319244 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3812260031700134 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6409465670585632 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 trainer/global_step‚4835 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16866710782051086 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145061194896698 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878019690513611 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3188701272010803 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5657961964607239 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4707951247692108 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993128776550293 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.384794145822525 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44929003715515137 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 6 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.305208295583725 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5228174328804016 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01305683609098196 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10168951749801636 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4041393995285034 jet/accuracy_4_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4357701539993286 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.553934633731842 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4972647726535797 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2770196795463562 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.316964328289032 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2939189076423645 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49763941764831543 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26045727729797363 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32357358932495117 jet/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017568625509738922 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_5‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5416666865348816 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 jet/accuracy_7_of_7‚0 *jet/accuracy_1_of_1‚0.4743501842021942 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010493827052414417 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01344086043536663 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2400883287191391 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5142543911933899 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2427636981010437 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1795911192893982 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1874934881925583 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48381659388542175 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 +jet/accuracy_2_of_3‚0.23637263476848602 ,particle/sensitivity‚0.33933767676353455 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37440529465675354 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2546793222427368 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.235904261469841 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9164262413978577 particle/accuracy_1_of_2‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5833333134651184 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.361477255821228 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.45258617401123047 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16067685186862946 particle/accuracy_4_of_5‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.039168696850538254 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7validation_average_jet_accuracy‚0.42751172184944153 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_5‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 (loss/val_total_loss‚42.9665242544926 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5142301321029663 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_3‚0.8536666035652161 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04386266693472862 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28124096989631653 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0021547081414610147 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001925770309753716 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.405773401260376 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1666666567325592 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2034391611814499 +jet/accuracy_3_of_4‚0.04727313667535782 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3212699890136719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5015483498573303 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5105769634246826 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10252950340509415 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151262283325 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998374879360199 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14179480075836182 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5269356369972229 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4596881866455078 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5241042971611023 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38771212100982666 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09777715057134628 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015786785632371902 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18463698029518127 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17624524235725403 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976100325584412 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1919201761484146 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4854651093482971 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2656139135360718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933074116706848 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41395828127861023 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3780324161052704 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2575666308403015 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17641571164131165 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26181960105895996 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 jet/accuracy_3_of_7‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32292765378952026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928156137466431 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20362438261508942 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903312921524048 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46135321259498596 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008773582987487316 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9805682897567749 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15132707357406616 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44392913579940796 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2074681520462036 +particle/specificity‚0.9540387392044067 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4627689719200134 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 % _timestamp‚1.7885304167263265e+09 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19851092994213104 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42696455121040344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1795911192893982 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4017130732536316 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.41352587938308716 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 _runtime‚5800.264064761 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4920729696750641 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925916790962219 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4631625711917877 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16081872582435608 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024636900052428246 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19276896119117737 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4587874412536621 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.36664122343063354 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46062880754470825 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 +jet/accuracy_3_of_5‚0.1247695833444!æ2b 5953 particle/accuracy_1_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 _step‚395 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5034587979316711 particle/accuracy_4_of_6‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22108779847621918 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03874470666050911 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21573156118392944 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.490875780582428 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_3‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 epoch‚3 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23084580898284912 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18312759697437286 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_5‚0.9537698030471802 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930601716041565 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02291666530072689 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.41734346747398376 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_1‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05I½vœÊ¥ :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10168951749801636 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4041393995285034 jet/accuracy_4_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4357701539993286 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.553934633731842 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4972647726535797 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2770196795463562 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.316964328289032 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2939189076423645 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49763941764831543 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26045727729797363 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32357358932495117 jet/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017568625509738922 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_5‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5416666865348816 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 jet/accuracy_7_of_7‚0 *jet/accuracy_1_of_1‚0.4743501842021942 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010493827052414417 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01344086043536663 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2400883287191391 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5142543911933899 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2427636981010437 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1795911192893982 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1874934881925583 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48381659388542175 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 +jet/accuracy_2_of_3‚0.23637263476848602 ,particle/sensitivity‚0.33933767676353455 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37440529465675354 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2546793222427368 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.235904261469841 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9164262413978577 particle/accuracy_1_of_2‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5833333134651184 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.361477255821228 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.45258617401123047 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16067685186862946 particle/accuracy_4_of_5‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.039168696850538254 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7validation_average_jet_accuracy‚0.42751172184944153 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_5‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 (loss/val_total_loss‚42.9665242544926 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5142301321029663 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_3‚0.8536666035652161 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04386266693472862 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28124096989631653 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0021547081414610147 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001925770309753716 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.405773401260376 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1666666567325592 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2034391611814499 +jet/accuracy_3_of_4‚0.04727313667535782 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3212699890136719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5015483498573303 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5105769634246826 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10252950340509415 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151262283325 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998374879360199 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14179480075836182 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5269356369972229 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4596881866455078 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5241042971611023 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38771212100982666 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09777715057134628 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015786785632371902 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18463698029518127 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17624524235725403 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976100325584412 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1919201761484146 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4854651093482971 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2656139135360718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933074116706848 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41395828127861023 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3780324161052704 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2575666308403015 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17641571164131165 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26181960105895996 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 jet/accuracy_3_of_7‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32292765378952026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928156137466431 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20362438261508942 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903312921524048 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46135321259498596 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008773582987487316 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9805682897567749 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15132707357406616 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44392913579940796 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2074681520462036 +particle/specificity‚0.9540387392044067 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4627689719200134 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 % _timestamp‚1.7885304167263265e+09 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19851092994213104 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42696455121040344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1795911192893982 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4017130732536316 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.41352587938308716 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 _runtime‚5800.264064761 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4920729696750641 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925916790962219 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4631625711917877 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16081872582435608 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024636900052428246 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19276896119117737 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009410430677235126 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4587874412536621 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.36664122343063354 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46062880754470825 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 +jet/accuracy_3_of_5‚0.12476958334445953 particle/accuracy_1_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 _step‚395 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5034587979316711 particle/accuracy_4_of_6‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22108779847621918 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03874470666050911 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21573156118392944 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.490875780582428 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_3‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 epoch‚3 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23084580898284912 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18312759697437286 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_5‚0.9537698030471802 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930601716041565 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02291666530072689 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.41734346747398376 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_1‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5295992493629456 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893428087234497 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1537792831659317 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962748885154724 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18857352435588837 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993098378181458 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.37914225459098816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2763386070728302 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24885372817516327 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0055158245377242565 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18936915695667267 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41853317618370056 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9755746722221375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5100612044334412 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007286648731678724 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.34090909361839294 particle/accuracy_2_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5052679181098938 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4748897850513458 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/f_score‚0.42730259895324707 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2924995720386505 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32309505343437195 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20641084015369415 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4474544823169708 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4042319655418396 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1711111217737198 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46699419617652893 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3632037341594696 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333432674408 +jet/accuracy_2_of_2‚0.10400119423866272 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6824390292167664 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301887214183807 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.36438480019569397 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4876176118850708 +jet/accuracy_2_of_5‚0.40844854712486267 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_2_of_7‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19619466364383698 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2011563926935196 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4852164387702942 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17708133161067963 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18779945373535156 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010673235170543194 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022399630397558212 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2020440399646759 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8224474787712097 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2626262307167053 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5438172817230225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4707951247692108 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.537869393825531 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5545138120651245 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4536789357662201 (particle/accuracy‚0.8580521941184998 jet/accuracy_5_of_6‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23055556416511536 +jet/accuracy_2_of_6‚0.49358975887298584 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9856238961219788 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5373322367668152 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42988502979278564 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32685184478759766 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902342557907104 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2330024689435959 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4830748736858368 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRä|«At2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.2666666805744171 particle/accuracy_6_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48749619722366333 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5310250520706177 particle/accuracy_2_of_2‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020364467054605484 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21423786878585815 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919083118438721 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984030008316 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4710519015789032 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.548214316368103 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7459497451782227 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01994902268052101 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3259672522544861 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 jet/accuracy_6_of_6‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.44290122389793396 particle/accuracy_3_of_4‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909665584564209 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19722366333007812 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023085733875632286 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880220293998718 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.18653270602226257 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19173391163349152 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3741742968559265 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2568965256214142 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08289338648319244 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3812260031700134 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6409465670585632 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 trainer/global_step‚4835 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16866710782051086 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145061194896698 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878019690513611 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3188701272010803 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5657961964607239 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4707951247692108 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993128776550293 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.384794145822525 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44929003715515137 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 6 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.305208295583725 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5228174328804016 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01305683609098196 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942‹D€Hõ‘‚(:† õëÔõÙ¯ß gpu.0.gpu‚6.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o8¿Pž‚(:Å õëÔÁíåß disk.sda2.in‚128.625! disk.sda2.out‚1106.6044921875! disk.sdb2.out‚1106.6044921875 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 cpu‚25.23911108821849 proc.cpu.threads‚17 network.sent‚ 1.2210141e+07 disk.md0.in‚ 199.5859375" disk.nvme0n1.out‚ 66778.6796875 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 disk.md0.out‚ 1083.2890625( proc.memory.availableMB‚ 1324.3359375 network.recv‚ 1.979817e+06 disk.sda1.out‚0 disk.nvme1n1.in‚0 disk.md0p1.out‚ 1083.2890625# proc.memory.rssMB‚ 3635.01953125) disk./.usagePercent‚75.95281762567853% disk./.usageGB‚2716.4406547546387 disk.sda1.in‚0 disk.sda.in‚128.625 disk.sda.out‚1106.6044921875! disk.nvme1n1.out‚ 23275.953125 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.rbd2.in‚0$ memory_percent‚91.91689491271973) proc.memory.percent‚22.18639850616455 disk.dm-0.in‚0 disk.dm-0.out‚ 90054.6328125 disk.sdb.out‚1106.6044921875 disk.rbd0.in‚0½¸–锟‚(:‰ „žëÔ–€óÞ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oܯE ‚(:º „žëÔ€÷ß disk.sda1.in‚0# proc.memory.rssMB‚ 2984.33203125 disk.md0p1.in‚ 199.5859375 disk.sda2.in‚128.625 disk.sda2.out‚ 1107.91015625 disk.md0.in‚ 199.5859375 cpu‚25.23849602214333 disk.md0p1.out‚ 1084.53125 disk.sda1.out‚0' proc.memory.availableMB‚ 1975.078125 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sdb.out‚ 1107.91015625 disk.sda.in‚128.625 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.62109375 network.sent‚ 1.2333477e+07$ disk./.usageGB‚2716.438201904297 disk.rbd2.in‚0 disk.nvme0n1.in‚0# memory_percent‚87.9450798034668 network.recv‚ 1.985369e+06 disk.sdb2.out‚ 1107.91015625 disk.sda.out‚ 1107.91015625 disk.nvme1n1.in‚0 disk.md0.out‚ 1084.53125) disk./.usagePercent‚75.95274904295638 disk.sdb2.in‚ 70.9609375" disk.nvme1n1.out‚ 23341.2578125# disk.nvme0n1.out‚68344.44921875* proc.memory.percent‚18.214917182922363 disk.sdb.in‚ 70.9609375 disk.dm-0.out‚91685.70703125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚01WŒ2¡- _wandbruntime‚5820  _runtime‚5820ÉÆ¢À & lr-AdamW/pg1‚0.0005476589498453851 & lr-AdamW/pg2‚0.0005476589498453851 trainer/global_step‚4849 $ _timestamp‚1.788530436479026e+09 _runtime‚5820.014636124 _step‚396×wÖˣŠ& lr-AdamW/pg1‚0.0005476589498453851 & lr-AdamW/pg2‚0.0005476589498453851 trainer/global_step‚4849 $ _timestamp‚1.788530436479026e+09 _runtime‚5820.014636124 _step‚396Œ­‰üS2¤- _wandbruntime‚5820  _runtime‚5820QXÄJú'¥ôO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122973859 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415113777324032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409550153611776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90508197 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 trainer/global_step‚4849 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29575674 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409550171614720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401508295881506 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101686746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395964563699952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415114705298432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5815346816000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543732181554 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409550153611776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122973859 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5815120131072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101686744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395964546660592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 trainer/global_step‚4849 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29575674 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409550171614720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401508295881506 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101686746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395964563699952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415114705298432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5815346816000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543732181554 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409550153611776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122973859 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5815120131072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101686744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395964546660592 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 6¹= z!I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101706906 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101706904 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90525347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401587505734964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415197445857792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5816289032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5565655157248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409631790700544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224705594 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101706906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _step‚398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29581992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29581360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457701850108928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185²¬Z '©ƒN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101706906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _step‚398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29581992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29581360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457701850108928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5816519898112 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101706906 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101706904 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90525347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401587505734964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415197445857792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5816289032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5565655157248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409631790700544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224705594 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185Ž‹Iu2ª- _wandbruntime‚5821  _runtime‚5821M¥\ûY«Ó 1loss/SRqqt4/assignment_loss‚1.324887923747301 epoch‚4 trainer/global_step‚4849 _step‚399 0loss/FRt4/assignment_loss‚2.4721039257398467 /loss/SRqqt2/detection_loss‚2.00896239687096 2loss/SRqqt1/assignment_loss‚1.7306473690729876 2loss/SRqqt2/assignment_loss‚1.4207649913362481 .loss/FRt1/detection_loss‚2.953444173506328 /loss/FBt1/detection_loss‚1.8614853645364444 /loss/FBt3/detection_loss‚0.6605294006935591 &loss/total_loss‚42.586772266454076 0loss/FRt1/assignment_loss‚3.3122416394097463 1loss/FBt2/assignment_loss‚0.27134580779862255 1loss/SRqqt4/detection_loss‚1.8407850098609924 /loss/FBt2/detection_loss‚3.3305516072681973 0loss/FRt3/assignment_loss‚1.2167221816147076 .loss/FRt2/detection_loss‚2.477795540283972 .loss/FRt4/detection_loss‚2.694119984874638 0loss/SRqqt1/detection_loss‚2.508959401112336 .loss/FBt4/detection_loss‚2.455891340970993 0loss/FRt2/assignment_loss‚3.5415949739258865 0loss/FBt1/assignment_loss‚0.2136701977063543 /loss/FRt3/detection_loss‚1.5489865475169875 1loss/SRqqt3/detection_loss‚0.7739657526835799 % _timestamp‚1.7885304375895083e+09 1loss/SRqqt3/assignment_loss‚0.997129006787788 1loss/FBt3/assignment_loss‚0.08258529138236877 0loss/FBt4/assignment_loss‚0.8876024377532303 _runtime‚5821.124781946Îï ^¬Ø .loss/FRt4/detection_loss‚2.694119984874638 0loss/SRqqt1/detection_loss‚2.508959401112336 .loss/FBt4/detection_loss‚2.455891340970993 0loss/FRt2/assignment_loss‚3.5415949739258865 0loss/FBt1/assignment_loss‚0.2136701977063543 /loss/FRt3/detection_loss‚1.5489865475169875 1loss/SRqqt3/detection_loss‚0.7739657526835799 % _timestamp‚1.7885304375895083e+09 1loss/SRqqt3/assignment_loss‚0.997129006787788 1loss/FBt3/assignment_loss‚0.08258529138236877 0loss/FBt4/assignment_loss‚0.8876024377532303 _runtime‚5821.124781946 1loss/SRqqt4/assignment_loss‚1.324887923747301 epoch‚4 trainer/global_step‚4849 _step‚399 0loss/FRt4/assignment_loss‚2.4721039257398467 /loss/SRqqt2/detection_loss‚2.00896239687096 2loss/SRqqt1/assignment_loss‚1.7306473690729876 2loss/SRqqt2/assignment_loss‚1.4207649913362481 .loss/FRt1/detection_loss‚2.953444173506328 /loss/FBt1/detection_loss‚1.8614853645364444 /loss/FBt3/detection_loss‚0.6605294006935591 &loss/total_loss‚42.586772266454076 0loss/FRt1/assignment_loss‚3.3122416394097463 1loss/FBt2/assignment_loss‚0.27134580779862255 1loss/SRqqt4/detection_loss‚1.8407850098609924 /loss/FBt2/detection_loss‚3.3305516072681973 0loss/FRt3/assignment_loss‚1.2167221816147076 .loss/FRt2/detection_loss‚2.477795540283972¼"´’­‚(:‡ “žëÔ’õ®à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uOW®‚(:Ì “žëÔúÒÄà disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚23406.75390625 disk.md0p1.out‚ 1085.95703125! disk.sda.out‚1109.38623046875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 91908.5703125 disk.sdb2.in‚ 70.9609375 cpu‚25.23787996504918 network.recv‚ 1.989164e+06$ disk./.usageGB‚2716.438148498535 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625" disk.sda2.out‚1109.38623046875$ memory_percent‚87.94193267822266# disk.nvme0n1.out‚68501.81640625 network.sent‚ 1.2351612e+07) disk./.usagePercent‚75.95274754970893" disk.sdb2.out‚1109.38623046875 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1109.38623046875 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.out‚ 1085.95703125& proc.memory.availableMB‚ 1975.59375 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375†´Ò÷“¯‚(:ˆ ¢žëÔ¹¼ôà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¬ ·R°‚(:Ç ¢žëÔ´–œâ disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17 network.recv‚ 1.993982e+06 disk.rbd0.out‚ 1.68359375 disk.sda1.out‚0 network.sent‚ 1.2445731e+07 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1087.6953125 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0) disk./.usagePercent‚75.95274754970893 disk.sda2.in‚128.625" disk.sda2.out‚1111.17431640625 disk.dm-0.out‚ 92196.1796875! disk.sdb.out‚1111.17431640625 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚87.94198036193848 disk.sda.in‚128.625# proc.memory.rssMB‚ 2984.33203125$ disk./.usageGB‚2716.438148498535" disk.nvme0n1.out‚ 68659.0546875" disk.sdb2.out‚1111.17431640625 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1111.17431640625 disk.nvme1n1.out‚ 23537.125 disk.md0p1.out‚ 1087.6953125 disk.rbd2.out‚ 1.62109375( proc.memory.availableMB‚ 1975.5859375 disk.sda1.in‚0 cpu‚25.237257300157076 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0!PŸ“±‚(:ˆ ±žëÔå‹ãà gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.822' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.2055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðD»0²‚(:¥ ±žëÔ¨›ãá) disk./.usagePercent‚75.95274648310361 disk.dm-0.in‚0 disk.sdb2.out‚1112.75 disk.sda.out‚1112.75 disk.rbd0.out‚ 1.68359375) proc.memory.availableMB‚ 1975.33984375 network.recv‚ 1.998848e+06 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚87.94348239898682 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚ 1.62109375# disk.nvme0n1.out‚68659.41796875 disk.md0.out‚ 1089.21484375 disk.sda.in‚128.625# disk.nvme1n1.out‚23602.58203125 disk.rbd0.in‚0 disk.sda2.out‚1112.75 disk.dm-0.out‚92262 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚18.214917182922363% disk./.usageGB‚2716.4381103515625 disk.rbd2.in‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 cpu‚25.236657078864706 disk.nvme0n1.in‚0 disk.sdb.out‚1112.75 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1089.21484375# proc.memory.rssMB‚ 2984.33203125 network.sent‚ 1.2451956e+07«ß=L2³- _wandbruntime‚5876  _runtime‚5876Ëç Ç´Á % _timestamp‚1.7885304926633663e+09 _runtime‚5876.198923981 _step‚400 & lr-AdamW/pg1‚0.0005430557873605745 & lr-AdamW/pg2‚0.0005430557873605745 trainer/global_step‚4899™ÌµÆ & lr-AdamW/pg1‚0.0005430557873605745 & lr-AdamW/pg2‚0.0005430557873605745 trainer/global_step‚4899 % _timestamp‚1.7885304926633663e+09 _runtime‚5876.198923981 _step‚4007F–“2¶- _wandbruntime‚5876  _runtime‚5876U€ûUœ·ôO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102694746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413631125905920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456306389728768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91397542 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5619711137280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419251747014656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419251747014656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873590066176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226913505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61535174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456306365788672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399915443427152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419250819040256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102694744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102694744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462179727072768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4899 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413631107902976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124218761 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29861755 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405514127849450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399915426387792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124218761 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚5876.199783365 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413631125905920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124245105 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5619711137280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124245105 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419250819040256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61535172 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462179979794944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226939851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5599609436662 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570í¤ÿ'¸ùO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29862368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405515052863814 % _timestamp‚1.7885304926641147e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91396927 _step‚401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413631107902976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124218761 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29861755 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405514127849450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399915426387792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124218761 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚5876.199783365 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413631125905920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124245105 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5619711137280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124245105 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419250819040256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61535172 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462179979794944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226939851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5599609436662 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102694746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413631125905920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456306389728768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91397542 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5619711137280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419251747014656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419251747014656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873590066176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226913505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61535174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456306365788672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399915443427152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419250819040256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102694744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102694744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462179727072768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4899 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405594262717272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102714906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124243592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91414778 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5620832597504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5600726710148 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269962 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102714904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91415401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5621742582272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456396615433216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456396591493120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419334487574016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462271374440448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5620832597504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29868162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5599818734936 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399994461021696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419333559586304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413712744991744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413712726988800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102714906 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462271121731584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29867541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413712744991744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0v«e>K!»‚N \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5600726710148 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269962 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102714904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91415401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5621742582272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456396615433216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456396591493120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419334487574016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462271374440448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5620832597504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29868162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5599818734936 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399994461021696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419333559586304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413712744991744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413712726988800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102714906 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462271121731584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29867541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413712744991744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788530493612704e+09 _runtime‚5877.148919845 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405594262717272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102714906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124243592 K>DeviceStatsMonitor.on_train_batch_end/inactive_"Eè ½split.all.freed‚91414778 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5620832597504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623’žxÐå2¼- _wandbruntime‚5877  _runtime‚5877Š1hZ½Ô 1loss/SRqqt2/detection_loss‚2.2188132114899464 _runtime‚5877.150669633 0loss/SRqqt1/detection_loss‚2.345951204498609 0loss/FRt4/assignment_loss‚2.7399895255630082 1loss/FBt3/assignment_loss‚0.08436828295367833 .loss/FRt1/detection_loss‚3.204037475174871 0loss/SRqqt3/detection_loss‚0.718884294041816 0loss/FRt1/assignment_loss‚3.3439019897888445 0loss/FRt2/assignment_loss‚3.6475759856402874 0loss/FRt3/assignment_loss‚1.1949724474949623 /loss/FRt2/detection_loss‚2.1794899434140547 1loss/SRqqt4/detection_loss‚1.9873561225831509 2loss/SRqqt3/assignment_loss‚1.0901891047573264 1loss/FBt1/assignment_loss‚0.07783447977271862 .loss/FRt3/detection_loss‚1.415024361503658 /loss/FBt4/detection_loss‚3.0834159016609193 _step‚403 0loss/SRqqt1/assignment_loss‚2.05386901833117 2loss/SRqqt2/assignment_loss‚1.3668953177447503 2loss/SRqqt4/assignment_loss‚1.4419497812050395 1loss/FBt2/assignment_loss‚0.05222282796666554 /loss/FBt4/assignment_loss‚0.309327258169651 /loss/FBt2/detection_loss‚2.4700992902119956 %loss/total_loss‚42.93312620896705 trainer/global_step‚4899 /loss/FBt1/detection_loss‚2.9370450258255003 $ _timestamp‚1.788530493615393e+09 /loss/FRt4/detection_loss‚2.4076547275825932 /loss/FBt3/detection_loss‚0.5622586315918353 epoch‚4ç&6î_¾Ù /loss/FRt4/detection_loss‚2.4076547275825932 /loss/FBt3/detection_loss‚0.5622586315918353 epoch‚4 1loss/SRqqt2/detection_loss‚2.2188132114899464 _runtime‚5877.150669633 0loss/SRqqt1/detection_loss‚2.345951204498609 0loss/FRt4/assignment_loss‚2.7399895255630082 1loss/FBt3/assignment_loss‚0.08436828295367833 .loss/FRt1/detection_loss‚3.204037475174871 0loss/SRqqt3/detection_loss‚0.718884294041816 0loss/FRt1/assignment_loss‚3.3439019897888445 0loss/FRt2/assignment_loss‚3.6475759856402874 0loss/FRt3/assignment_loss‚1.1949724474949623 /loss/FRt2/detection_loss‚2.1794899434140547 1loss/SRqqt4/detection_loss‚1.9873561225831509 2loss/SRqqt3/assignment_loss‚1.0901891047573264 1loss/FBt1/assignment_loss‚0.07783447977271862 .loss/FRt3/detection_loss‚1.415024361503658 /loss/FBt4/detection_loss‚3.0834159016609193 _step‚403 0loss/SRqqt1/assignment_loss‚2.05386901833117 2loss/SRqqt2/assignment_loss‚1.3668953177447503 2loss/SRqqt4/assignment_loss‚1.4419497812050395 1loss/FBt2/assignment_loss‚0.05222282796666554 /loss/FBt4/assignment_loss‚0.309327258169651 /loss/FBt2/detection_loss‚2.4700992902119956 %loss/total_loss‚42.93312620896705 trainer/global_step‚4899 /loss/FBt1/detection_loss‚2.9370450258255003 $ _timestamp‚1.788530493615393e+09“Ùß’’¿‚(:‡ ÀžëÔùÁá gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆþÚTÀ‚(:É ÀžëÔëÀ‘â# disk./.usageGB‚2716.43892288208 disk.nvme0n1.in‚0 disk.md0.out‚ 1090.57421875 disk.dm-0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1114.1748046875 disk.sdb2.in‚ 70.9609375$ memory_percent‚87.94558048248291 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.62109375 cpu‚25.236011547408857# disk.nvme1n1.out‚23667.80078125 disk.md0p1.out‚ 1090.57421875 disk.sda.out‚1114.1748046875 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb2.out‚1114.1748046875 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.nvme0n1.out‚ 68816.65625! disk.sda2.out‚1114.1748046875 disk.md0.in‚ 199.5859375 disk.dm-0.out‚92484.45703125# proc.memory.rssMB‚ 2984.33203125 disk.nvme1n1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.2471021e+07 network.recv‚ 2.003331e+06) disk./.usagePercent‚75.95276920179694) proc.memory.availableMB‚ 1974.99609375* proc.memory.percent‚18.214917182922363 disk.sda2.in‚128.625 disk.sda.in‚128.625ì.墔Á‚(:‰ ÏžëÔ¿«Ìß gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o‘®S‚(:È ÏžëÔâŒÓá network.sent‚ 1.2564101e+07# disk.nvme0n1.out‚69026.41796875 disk.sda2.in‚128.625 disk.sda1.in‚0( proc.memory.availableMB‚ 1975.4765625# proc.memory.rssMB‚ 2984.33203125 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1116.0322265625 disk.sdb1.out‚0) disk./.usagePercent‚75.95276866849429 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚92759.58203125% disk./.usageGB‚2716.4389038085938 disk.rbd2.in‚0 disk.sdb.out‚1116.0322265625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1092.359375 network.recv‚ 2.007555e+06 disk.rbd2.out‚ 1.62109375 disk.nvme0n1.in‚0! disk.sda2.out‚1116.0322265625 disk.md0.out‚ 1092.359375 disk.sda.in‚128.625" disk.nvme1n1.out‚ 23733.1640625 disk.rbd0.out‚ 1.68359375! disk.sdb2.out‚1116.0322265625 disk.sdb1.in‚0* proc.memory.percent‚18.214917182922363 cpu‚25.235458740897155 disk.dm-0.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0$ memory_percent‚87.94264793395996 proc.cpu.threads‚17ñýl-”Â(:‰ ÞžëÔÌý¡à gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÞÙâ@Ä‚(:µ ÞžëÔòÊÔà network.sent‚ 1.2571916e+07 disk.rbd0.in‚0 disk.sda2.in‚128.625 disk.sda2.out‚ 1117.79296875 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚ 1117.79296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 199.5859375 disk.sdb2.out‚ 1117.79296875* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 cpu‚25.234914340361215 disk.rbd0.out‚ 1.68359375 disk.sda.in‚128.625 disk.dm-0.in‚0# proc.memory.rssMB‚ 2984.33203125 disk.md0p1.out‚ 1094.0625 disk.rbd2.in‚0 disk.nvme0n1.in‚0 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1094.0625) disk./.usagePercent‚75.95277026840228% disk./.usageGB‚2716.4389610290527 disk.sda.out‚ 1117.79296875 disk.nvme1n1.in‚0# disk.nvme1n1.out‚23863.44140625 disk.dm-0.out‚93047.09765625$ memory_percent‚89.61944580078125 network.recv‚ 2.011376e+06 disk.nvme0n1.out‚ 69183.65625# proc.memory.availableMB‚1700.75Ð ±}“Å‚(:ˆ ížëÔ†ê¡à gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÃóBWÆ‚(:Ì ížëÔ‡¾”á" disk.nvme0n1.out‚ 69340.9453125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625! disk.sda.out‚1119.31201171875 disk.nvme1n1.in‚0* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17% disk./.usageGB‚2716.4389610290527 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 93237.2109375 disk.sdb.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125 network.recv‚ 2.015852e+06" disk.sda2.out‚1119.31201171875" disk.sdb2.out‚1119.31201171875! disk.nvme1n1.out‚ 23896.265625 network.sent‚ 1.2578053e+07 disk.rbd2.out‚ 1.62109375 disk.dm-0.in‚0$ memory_percent‚92.62588024139404 disk.sdb1.out‚0! disk.sdb.out‚1119.31201171875) proc.memory.availableMB‚ 1208.17578125 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.234359608162947 disk.sdb1.in‚0) disk./.usagePercent‚75.95277026840228 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 1095.5234375 disk.md0p1.out‚ 1095.5234375 ›ŒZ2Ç- _wandbruntime‚5933  _runtime‚5933á5nÆÈÀ _runtime‚5933.755853448 _step‚404 & lr-AdamW/pg1‚0.0005384157058204934 & lr-AdamW/pg2‚0.0005384157058204934 trainer/global_step‚4949 $ _timestamp‚1.788530550220216e+09m‡ÇJËÉÅ & lr-AdamW/pg1‚0.0005384157058204934 & lr-AdamW/pg2‚0.0005384157058204934 trainer/global_step‚4949 $ _timestamp‚1.788530550220216e+09 _runtime‚5933.755853448 _step‚404”oÀ$2Ê- _wandbruntime‚5933  _runtime‚5933`U:Ûù'ËóO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417712062194176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125463663 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5676708533760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5931766567936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62138424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403866323154352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229166407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92289276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403866306114992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚405 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417712080197120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423387860756480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103702744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5676708533760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423388788730880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125490007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62138422 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466749417579008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423388788730880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92289890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409521809846122 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125490007 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚5933.757281434 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5675798562304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423387860756480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417712062194176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417712080197120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125463663 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5655486691770 trainer/global_step‚4949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466749674495488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712¢¨º×þ'ÌøO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5675798562304 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103702746 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30150854 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460817651011072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5931999544320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5676708533760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423388788730880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125490007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62138422 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466749417579008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423388788730880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92289890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409521809846122 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125490007 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚5933.757281434 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5675798562304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423387860756480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417712062194176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417712080197120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125463663 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5655486691770 trainer/global_step‚4949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466749674495488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417712062194176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125463663 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5676708533760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5931766567936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62138424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403866323154352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229166407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92289276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403866306114992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚405 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417712080197120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423387860756480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103702744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92307034 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229237770 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417793681280000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5933160715264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409601019699580 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5676920022528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885305512643752e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229237770 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229211398 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417793699282944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460907876715520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103722904 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423471529290240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5676920022528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423471529290240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460907900655616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30156547 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229211398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62150487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30157162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615=¼° 'σN XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5676920022528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885305512643752e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229237770 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229211398 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417793699282944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460907876715520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103722904 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423471529290240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5676920022528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423471529290240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460907900655616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30156547 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229211398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62150487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30157162 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417793699282944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103722906 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125514864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125514864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5656603965256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417793681280000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92307034 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229237770 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417793681280000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5933160715264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409601019699580 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0–»µy§2Ð- _wandbruntime‚5934  _runtime‚5934;5‹ZÑÔ /loss/FBt3/detection_loss‚0.7335713833770892 1loss/FBt4/assignment_loss‚0.24916164747588077 .loss/FBt2/detection_loss‚2.978138585885366 trainer/global_step‚4949 2loss/SRqqt2/assignment_loss‚1.3142934867434881 1loss/SRqqt3/assignment_loss‚1.016232302293268 /loss/FBt4/detection_loss‚2.7262014084392123 epoch‚4 /loss/FBt1/detection_loss‚2.3956472973028817 _runtime‚5934.802268733 .loss/FRt2/detection_loss‚2.193724521814381 /loss/FRt4/detection_loss‚2.2496721573478107 0loss/SRqqt1/detection_loss‚1.982640108891896 0loss/SRqqt4/detection_loss‚2.146168205000105 2loss/SRqqt1/assignment_loss‚1.5935644290276936 2loss/SRqqt4/assignment_loss‚1.4544905677792572 0loss/FBt3/assignment_loss‚0.1401516809296575 1loss/SRqqt3/detection_loss‚0.8484938425596716 &loss/total_loss‚40.354952437616475 /loss/FRt1/assignment_loss‚2.500636297113755 /loss/FRt3/detection_loss‚1.4999448532859485 1loss/SRqqt2/detection_loss‚1.9400875676761975 .loss/FRt1/detection_loss‚2.873868472435895 0loss/FRt3/assignment_loss‚1.3560058909749229 /loss/FRt4/assignment_loss‚2.881833682254869 0loss/FBt1/assignment_loss‚0.1720558613401598 2loss/FBt2/assignment_loss‚0.005778476996713127 % _timestamp‚1.7885305512669995e+09 _step‚407 /loss/FRt2/assignment_loss‚3.102589710670359 \ÉK_ÒÙ epoch‚4 /loss/FBt1/detection_loss‚2.3956472973028817 _runtime‚5934.802268733 .loss/FRt2/detection_loss‚2.193724521814381 /loss/FRt4/detection_loss‚2.2496721573478107 0loss/SRqqt1/detection_loss‚1.982640108891896 0loss/SRqqt4/detection_loss‚2.146168205000105 2loss/SRqqt1/assignment_loss‚1.5935644290276936 2loss/SRqqt4/assignment_loss‚1.4544905677792572 0loss/FBt3/assignment_loss‚0.1401516809296575 1loss/SRqqt3/detection_loss‚0.8484938425596716 &loss/total_loss‚40.354952437616475 /loss/FRt1/assignment_loss‚2.500636297113755 /loss/FRt3/detection_loss‚1.4999448532859485 1loss/SRqqt2/detection_loss‚1.9400875676761975 .loss/FRt1/detection_loss‚2.873868472435895 0loss/FRt3/assignment_loss‚1.3560058909749229 /loss/FRt4/assignment_loss‚2.881833682254869 0loss/FBt1/assignment_loss‚0.1720558613401598 2loss/FBt2/assignment_loss‚0.005778476996713127 % _timestamp‚1.7885305512669995e+09 _step‚407 /loss/FRt2/assignment_loss‚3.102589710670359 /loss/FBt3/detection_loss‚0.7335713833770892 1loss/FBt4/assignment_loss‚0.24916164747588077 .loss/FBt2/detection_loss‚2.978138585885366 trainer/global_step‚4949 2loss/SRqqt2/assignment_loss‚1.3142934867434881 1loss/SRqqt3/assignment_loss‚1.016232302293268 /loss/FBt4/detection_loss‚2.7262014084392123—™í¸É”Ó‚(:‰ üžëÔÍØÞ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¤PÔ‚(:Å üžëÔã–Ëá disk.rbd0.in‚0# disk.nvme0n1.out‚69498.18359375 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17 network.sent‚ 1.259484e+07 disk.sda1.out‚0 disk.nvme0n1.in‚0# memory_percent‚92.6274299621582' proc.memory.availableMB‚ 1207.921875 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1121.25244140625 disk.sda.in‚128.625 disk.rbd0.out‚ 1.68359375) disk./.usagePercent‚75.95277026840228 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1097.40625 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.214917182922363" disk.sda2.out‚1121.25244140625 disk.dm-0.out‚93524.73046875! disk.sda.out‚1121.25244140625 disk.nvme1n1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.233766413472015 disk.rbd2.in‚0 disk.rbd2.out‚ 1.62109375 disk.sda2.in‚128.625" disk.sdb2.out‚1121.25244140625 network.recv‚ 2.02025e+06% disk./.usageGB‚2716.4389610290527 disk.dm-0.in‚0! disk.nvme1n1.out‚ 24026.546875 disk.md0p1.out‚ 1097.40625½°“Õ‚(:ˆ ‹ŸëÔž‰ßÞ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£–ÐñPÖ‚(:Å ‹ŸëÔ±à¸á disk.sda1.out‚0) disk./.usagePercent‚75.93794786757438 disk.nvme0n1.in‚0 disk.sda.out‚1122.7451171875# disk.nvme0n1.out‚69655.60546875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 disk.nvme1n1.in‚0 cpu‚25.23313936898738 network.sent‚ 1.269135e+07 network.recv‚ 2.02546e+06 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375# memory_percent‚92.6259994506836 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.md0.out‚ 1098.84765625 disk.dm-0.in‚0 disk.sdb.out‚1122.7451171875# disk.nvme1n1.out‚24081.81640625% disk./.usageGB‚2715.9088401794434! disk.sda2.out‚1122.7451171875 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0& proc.memory.availableMB‚ 1208.15625 disk.sda2.in‚128.625 disk.dm-0.out‚ 93737.421875 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363 disk.rbd2.out‚ 1.62109375! disk.sdb2.out‚1122.7451171875 disk.md0p1.out‚ 1098.84765625q?Ñׂ֔(:‰ šŸëÔòØÿà gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†k #PØ‚(:Å šŸëÔƒžâ* proc.memory.percent‚18.214917182922363 network.recv‚ 2.02915e+06) disk./.usagePercent‚75.93793346840255 disk.md0p1.out‚ 1100.078125 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sdb.out‚1124.02099609375 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1124.02099609375 disk.sda1.out‚0" disk.sda2.out‚1124.02099609375! disk.sda.out‚1124.02099609375 cpu‚25.23259179434761 proc.cpu.threads‚17! disk.nvme1n1.out‚ 24081.984375 disk.nvme0n1.out‚ 69812.84375 disk.md0.out‚ 1100.078125 disk.dm-0.in‚0 network.sent‚ 1.2694812e+07 disk.nvme1n1.in‚0 disk.dm-0.out‚ 93894.828125# proc.memory.rssMB‚ 2984.33203125 disk.sda1.in‚0 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.rbd2.out‚ 1.62109375 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.68359375 disk.nvme0n1.in‚0 disk.md0.in‚ 199.5859375% disk./.usageGB‚2715.9083251953125 disk.rbd0.in‚0 disk.sda2.in‚128.625$ memory_percent‚92.62721538543701) proc.memory.availableMB‚ 1207.95703125ªîߔق(:‰ ©ŸëÔ•Æ¡à gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05;á#XÚ‚(:Í ©ŸëÔ±÷Þà disk.dm-0.in‚0 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17 disk.dm-0.out‚94087.07421875! disk.sdb.out‚1125.65283203125" disk.sdb2.out‚1125.65283203125! disk.sda.out‚1125.65283203125# proc.memory.rssMB‚ 2984.33203125 disk.rbd2.out‚ 1.62109375) disk./.usagePercent‚75.93871752997417! disk.nvme1n1.out‚ 24082.953125 disk.sda1.out‚0( proc.memory.availableMB‚ 1208.2109375* proc.memory.percent‚18.214917182922363 disk.sda.in‚128.625 disk.md0p1.out‚ 1101.66015625# disk.nvme0n1.out‚70004.12109375" disk.sda2.out‚1125.65283203125 network.sent‚ 1.2703201e+07 cpu‚25.23202327670352$ disk./.usageGB‚2715.936367034912 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 1101.66015625 network.recv‚ 2.033726e+06 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0$ memory_percent‚92.62566566467285`^>2Û- _wandbruntime‚5990  _runtime‚5990º¡7ÔÆÜÀ _step‚408 & lr-AdamW/pg1‚0.0005337396722111805 & lr-AdamW/pg2‚0.0005337396722111805 trainer/global_step‚4999 $ _timestamp‚1.788530607333456e+09 _runtime‚5990.869058296ˆéÔËÝÅ _runtime‚5990.869058296 _step‚408 & lr-AdamW/pg1‚0.0005337396722111805 & lr-AdamW/pg2‚0.0005337396722111805 trainer/global_step‚4999 $ _timestamp‚1.788530607333456e+09˜Œùõá2Þ- _wandbruntime‚5990  _runtime‚5990Z ã!ßôO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5731885987328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427524902472704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126708565 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990053212160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5731885987328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5732795958784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93181357 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62741672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407817185842192 _runtime‚5990.873200675 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421793016485376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30439685 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421793016485376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _step‚409 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427525830447104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62741674 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚407817202881552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126734909 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104710746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126734909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93181965 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421793034488320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126708565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30440291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471319248459264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471318989445632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885306073350844e+09 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231Œ]Bÿ'àùO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5731885987328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427524902472704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126708565 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990053212160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5731885987328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5732795958784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93181357 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62741672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407817185842192 _runtime‚5990.873200675 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421793016485376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30439685 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421793016485376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _step‚409 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427525830447104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62741674 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚407817202881552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126734909 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104710746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126734909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93181965 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421793034488320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126708565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30440291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471319248459264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471318989445632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885306073350844e+09 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640™<©:2á- _wandbruntime‚5991  _runtime‚5991ÃåÚÿ'âýM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5733917432320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126759766 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231464300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚5991.961806533 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93199077 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93199675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465419161937920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5711573245152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471410389797888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126733396 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407896203436736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421874653574144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚410 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231490672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407896220476096 trainer/global_step‚4999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421874635571200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231490672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62753737 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5733007447552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126759766 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30445340 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62753739 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465419185878016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413608701696460 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421874653574144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5733917432320 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104730906 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427607643018752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427608571006464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413607776681888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5712481220364 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788530608425618e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421874635571200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30445936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104730906 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427608571006464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5991454531584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104730904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006ªÝqÉÐã‚N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421874653574144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5733917432320 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104730906 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427607643018752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427608571006464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413607776681888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5712481220364 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788530608425618e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421874635571200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30445936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104730906 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427608571006464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5991454531584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104730904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5733917432320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126759766 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231464300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚5991.961806533 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93199077 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93199675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465419161937920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5711573245152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471410389797888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126733396 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407896203436736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421874653574144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚410 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231490672 ZMDeviceStatsMonitor.on_train_batch_end/iX«qY8 nactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407896220476096 trainer/global_step‚4999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421874635571200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231490672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62753737 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5733007447552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126759766 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30445340 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62753739 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465419185878016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413608701696460 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144šã¥å2ä- _wandbruntime‚5991  _runtime‚5991wpF]å× /loss/FRt4/detection_loss‚2.2327316741264145 1loss/SRqqt3/detection_loss‚0.8553800991839833 %loss/total_loss‚40.58902011570083 % _timestamp‚1.7885306084287665e+09 epoch‚4 /loss/FRt1/assignment_loss‚3.296619450344759 2loss/SRqqt2/assignment_loss‚1.3936774559804928 2loss/SRqqt4/assignment_loss‚1.3612198395297883 2loss/FBt1/assignment_loss‚0.017552153673022985 0loss/SRqqt1/detection_loss‚2.254688627579633 /loss/FBt3/detection_loss‚0.5025525709592499 /loss/FRt4/assignment_loss‚3.329425060471823 0loss/FBt3/assignment_loss‚0.0571784405222268 .loss/FRt1/detection_loss‚2.787666431244682 /loss/FRt3/detection_loss‚1.4000162239839782 1loss/SRqqt4/detection_loss‚1.9199730113465736 _step‚411 0loss/FRt2/assignment_loss‚3.3113191092715546 /loss/FBt2/detection_loss‚1.9126406013965607 _runtime‚5991.964415956 1loss/SRqqt2/detection_loss‚1.7346738563931507 .loss/FBt4/detection_loss‚2.431747430562973 1loss/FBt2/assignment_loss‚0.13059862687224758 /loss/FRt2/detection_loss‚2.6706910442424125 0loss/FRt3/assignment_loss‚1.0647247406973768 1loss/SRqqt1/assignment_loss‚1.880257305196103 1loss/FBt4/assignment_loss‚0.19799782143672928 trainer/global_step‚4999 2loss/SRqqt3/assignment_loss‚1.0051574597238666 /loss/FBt1/detection_loss‚2.8405310809612274±›ìbæÜ _runtime‚5991.964415956 1loss/SRqqt2/detection_loss‚1.7346738563931507 .loss/FBt4/detection_loss‚2.431747430562973 1loss/FBt2/assignment_loss‚0.13059862687224758 /loss/FRt2/detection_loss‚2.6706910442424125 0loss/FRt3/assignment_loss‚1.0647247406973768 1loss/SRqqt1/assignment_loss‚1.880257305196103 1loss/FBt4/assignment_loss‚0.19799782143672928 trainer/global_step‚4999 2loss/SRqqt3/assignment_loss‚1.0051574597238666 /loss/FBt1/detection_loss‚2.8405310809612274 /loss/FRt4/detection_loss‚2.2327316741264145 1loss/SRqqt3/detection_loss‚0.8553800991839833 %loss/total_loss‚40.58902011570083 % _timestamp‚1.7885306084287665e+09 epoch‚4 /loss/FRt1/assignment_loss‚3.296619450344759 2loss/SRqqt2/assignment_loss‚1.3936774559804928 2loss/SRqqt4/assignment_loss‚1.3612198395297883 2loss/FBt1/assignment_loss‚0.017552153673022985 0loss/SRqqt1/detection_loss‚2.254688627579633 /loss/FBt3/detection_loss‚0.5025525709592499 /loss/FRt4/assignment_loss‚3.329425060471823 0loss/FBt3/assignment_loss‚0.0571784405222268 .loss/FRt1/detection_loss‚2.787666431244682 /loss/FRt3/detection_loss‚1.4000162239839782 1loss/SRqqt4/detection_loss‚1.9199730113465736 _step‚411 0loss/FRt2/assignment_loss‚3.3113191092715546 /loss/FBt2/detection_loss‚1.9126406013965607›ÐW_¹”ç‚(:‰ ¸ŸëÔµãåß gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜™Sè‚(:È ¸ŸëÔÌÖ€â) disk./.usagePercent‚75.93879251232825# disk./.usageGB‚2715.93904876709 disk.sda1.in‚0! disk.sda2.out‚1127.4560546875 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125# disk.nvme1n1.out‚24083.16015625 network.sent‚ 1.2707186e+07 disk.sda2.in‚128.625! disk.sdb2.out‚1127.4560546875 disk.md0p1.out‚ 1103.3984375" disk.nvme0n1.out‚ 70161.5078125 disk.rbd2.out‚ 1.62109375 disk.sdb.out‚1127.4560546875 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1103.3984375$ memory_percent‚92.62750148773193* proc.memory.percent‚18.214917182922363 disk.sda.in‚128.625 network.recv‚ 2.038428e+06 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚94244.66796875 cpu‚25.23147598460165 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚1127.4560546875) proc.memory.availableMB‚ 1207.91015625 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.rbd2.in‚0552í”é‚(:‰ ÇŸëÔÂûßÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý0)«_ê‚(:Ô ÇŸëÔÕîÜá" disk.nvme0n1.out‚ 70406.6171875( proc.memory.availableMB‚ 1207.8984375 disk.md0.out‚1104.875 disk.sdb.out‚1128.990234375 disk.sdb2.out‚1128.990234375* proc.memory.percent‚18.214917182922363 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚94490.29296875$ memory_percent‚92.62757301330566 disk.sda2.in‚128.625% disk./.usageGB‚2716.0642280578613 disk.rbd2.out‚ 1.63671875 disk.nvme0n1.in‚0.609375 disk.dm-0.in‚ 8.2890625 disk.md0p1.in‚ 199.5859375 network.recv‚ 2.043244e+06 disk.rbd0.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.out‚1104.875 cpu‚25.230861949419396 network.sent‚ 1.2816273e+07) disk./.usagePercent‚75.94229257768983 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.out‚1128.990234375# disk.nvme1n1.out‚24083.67578125# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.out‚ 1.7578125 disk.sda.in‚128.625 disk.sda.out‚1128.990234375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.in‚0pǸüXë‚(:Í ÖŸëÔ·Éá disk.dm-0.out‚94555.56640625 disk.md0p1.out‚ 1106.23828125 disk.md0.in‚ 199.5859375 network.recv‚ 2.047352e+06 disk.nvme0n1.in‚0.609375 disk.sda2.out‚ 1130.40625 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1130.40625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb1.in‚0 cpu‚25.230291312138302) disk./.usagePercent‚75.94579456294099% disk./.usageGB‚2716.1894760131836 disk.rbd2.out‚ 1.69921875 disk.sda.out‚ 1130.40625# disk.nvme1n1.out‚24083.86328125 disk.sda1.out‚0# proc.memory.rssMB‚ 2984.33203125 disk.sdb.out‚ 1130.40625 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.7578125 disk.md0.out‚ 1106.23828125 disk.rbd2.in‚0 disk.nvme1n1.in‚ 7.6796875$ memory_percent‚92.62604713439941 network.sent‚ 1.2821968e+07* proc.memory.percent‚18.214917182922363 disk.sdb1.out‚0 disk.dm-0.in‚ 8.2890625 disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.sda1.in‚0( proc.memory.availableMB‚ 1208.1484375! disk.nvme0n1.out‚ 70471.703125¢€„c”ì‚(:‰ ÖŸëÔú Üá gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06?·Ö“í‚(:ˆ åŸëÔƒŸ£æ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙI3*mî‚(:â åŸëÔŒ“²é disk.rbd2.out‚ 1.69921875 disk.nvme0n1.in‚0.609375 disk.md0p1.out‚ 1108.046875# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1132.2666015625 disk.sda1.out‚0 disk.sda2.in‚128.625! disk.sda2.out‚1132.2666015625 disk.md0.out‚ 1108.046875 disk.sdb.out‚1132.2666015625 disk.rbd2.in‚0 network.sent‚ 1.2825906e+07% disk./.usageGB‚2716.1889114379883 disk.sda.out‚1132.2666015625" disk.nvme1n1.out‚ 24084.1953125 disk.rbd0.in‚0 disk.sdb1.in‚0 cpu‚25.229738589142556 disk.dm-0.out‚94830.92578125 disk.rbd0.out‚ 1.8369140625 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.94577877718223# disk.nvme0n1.out‚70746.73046875 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875$ memory_percent‚92.62521266937256 network.recv‚ 2.051886e+06 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0) proc.memory.availableMB‚ 1208.28515625 disk.dm-0.in‚ 8.2890625[õ‰W2ï- _wandbruntime‚6048  _runtime‚6048dKhhÆðÀ & lr-AdamW/pg2‚0.0005290286610110309 trainer/global_step‚5049 % _timestamp‚1.7885306649367025e+09 _runtime‚ 6048.47224846 _step‚412 & lr-AdamW/pg1‚0.0005290286610110309³müËñÅ & lr-AdamW/pg1‚0.0005290286610110309 & lr-AdamW/pg2‚0.0005290286610110309 trainer/global_step‚5049 % _timestamp‚1.7885306649367025e+09 _runtime‚ 6048.47224846 _step‚412œ¤Ýz±2ò- _wandbruntime‚6048  _runtime‚6048ü'QÚú'óôO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127953467 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425873988779520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411768082608752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469840245395968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63344922 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6048505390592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30728334 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475888981665792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885306649380155e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431661944188928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105718744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5788883383808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127953467 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475888726846464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417534398796374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431662872163328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5788883383808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127979811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6048736269824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425873970776576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105718746 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425873988779520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431661944188928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105718746 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5767241201986 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417535323810738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚4698402214558721©C/¤&ôùO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475888726846464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417534398796374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431662872163328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5788883383808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127979811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6048736269824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425873970776576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105718746 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425873988779520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431661944188928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105718746 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5767241201986 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417535323810738 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚469840221455872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127953467 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425873988779520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411768082608752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469840245395968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63344922 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6048505390592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30728334 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475888981665792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885306649380155e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431661944188928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105718744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allg îu[ocated‚ 5788883383808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127953467 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44p¹2õ- _wandbruntime‚6049  _runtime‚6049\d‰'öýM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30734152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475980382283264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 _runtime‚ 6049.50247309 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5768358475472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127978298 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94090523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475980133768704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411847083163936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105738906 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431745612722688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63356989 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425955607865344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94091141 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105738904 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128004668 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233717202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431745612722688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5789094872576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105738904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431744684734976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚469930471100416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233743574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30733536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417614533664196 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411847100203296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105738906 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640Óëä‰'÷‚N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63356989 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425955607865344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94091141 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105738904 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128004668 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233717202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431745612722688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5789094872576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105738904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431744684734976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚469930471100416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233743574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30733536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417614533664196 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411847100203296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105738906 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30734152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475980382283264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 _runtime‚ 6049.50247309 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5768358475472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127978298 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94090523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475980133768704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411847083163936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105738906 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431745612722688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31017668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435798985905152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106726746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129224713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435798985905152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106726746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5844970808832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5844060837376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5822210482090 _step‚417 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435799913879552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5844060837376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5844970808832 % _timestamp‚1.7885307225367057e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129224713 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235925113 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31017038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429954925067776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474351530618368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415718945296592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480459014867968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63948174 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6107484249600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚6106.072314742 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31017668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435798985905152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106726746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129224713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435798985905152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106726746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5844970808832 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5844060837376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5822210482090 _step‚417 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435799913879552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5844060837376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5844970808832 % _timestamp‚1.7885307225367057e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129224713 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235925113 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31017038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429954925067776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474351530618368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415718945296592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480459014867968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63948174 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6107484249600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚6106.072314742 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¡£ITí2‰- _wandbruntime‚6107  _runtime‚6107¬|›I'ŠþM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435882654438912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94982969 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480550404721152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480550152012288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415797962891136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5846092282368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106746904 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885307234854577e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421621290646504 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430036544153600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129223200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435882654438912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106746904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5823327755368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235970104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5824235730580 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129249570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106746906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5846092282368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6108648398336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚421622215661076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31022732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5845182297600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106746906 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129249570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63960239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474441756322816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474441732382720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235970104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430036562156544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430036544153600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚6107.021563503 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415797979930496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚6107.021563503 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415797979930496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94982969 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480550404721152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480550152012288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415797962891136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5846092282368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106746904 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885307234854577e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421621290646504 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430036544153600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129223200 Tt“Sr@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435882654438912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106746904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5823327755368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235970104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5824235730580 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129249570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106746906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5846092282368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6108648398336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚421622215661076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31022732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5845182297600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106746906 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129249570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63960239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474441756322816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474441732382720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235970104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430036562156544¢?Ðÿ22Œ- _wandbruntime‚6107  _runtime‚6107+‹.YÓ 0loss/FBt3/detection_loss‚0.38364639278087354 _runtime‚ 6107.0230607 /loss/FRt4/assignment_loss‚3.422681831500747 2loss/SRqqt3/assignment_loss‚0.8994625117629766 2loss/SRqqt4/assignment_loss‚1.4349499110960298 1loss/FBt4/assignment_loss‚0.07855900559887471 .loss/FRt3/detection_loss‚1.470222537686664 /loss/FRt4/detection_loss‚2.3887676481496203 2loss/SRqqt1/assignment_loss‚1.3415359806801592 2loss/SRqqt2/assignment_loss‚1.1822113220890362 0loss/FBt1/assignment_loss‚0.3220077807081301 0loss/SRqqt1/detection_loss‚2.748024338767642 epoch‚4 trainer/global_step‚5099 /loss/FRt2/assignment_loss‚3.205152544482001 /loss/FRt1/detection_loss‚3.1580191487851352 0loss/SRqqt4/detection_loss‚2.094087830020322 /loss/FBt1/detection_loss‚2.4822264194488524 $loss/total_loss‚43.5680715237713 _step‚419 0loss/FRt1/assignment_loss‚3.8150346590125044 0loss/FRt3/assignment_loss‚1.6556937817543271 1loss/SRqqt2/detection_loss‚2.5342237011150077 1loss/FBt2/assignment_loss‚0.30844344547937225 1loss/FBt3/assignment_loss‚0.08994140121109634 /loss/FRt2/detection_loss‚2.7406468351622904 1loss/SRqqt3/detection_loss‚0.9210437341141946 % _timestamp‚1.7885307234877827e+09 .loss/FBt2/detection_loss‚2.622608944773674 .loss/FBt4/detection_loss‚2.2688798175917737}Ѭ^ŽØ 0loss/FBt3/detection_loss‚0.38364639278087354 _runtime‚ 6107.0230607 /loss/FRt4/assignment_loss‚3.422681831500747 2loss/SRqqt3/assignment_loss‚0.8994625117629766 2loss/SRqqt4/assignment_loss‚1.4349499110960298 1loss/FBt4/assignment_loss‚0.07855900559887471 .loss/FRt3/detection_loss‚1.470222537686664 /loss/FRt4/detection_loss‚2.3887676481496203 2loss/SRqqt1/assignment_loss‚1.3415359806801592 2loss/SRqqt2/assignment_loss‚1.1822113220890362 0loss/FBt1/assignment_loss‚0.3220077807081301 0loss/SRqqt1/detection_loss‚2.748024338767642 epoch‚4 trainer/global_step‚5099 /loss/FRt2/assignment_loss‚3.205152544482001 /loss/FRt1/detection_loss‚3.1580191487851352 0loss/SRqqt4/detection_loss‚2.094087830020322 /loss/FBt1/detection_loss‚2.4822264194488524 $loss/total_loss‚43.5680715237713 _step‚419 0loss/FRt1/assignment_loss‚3.8150346590125044 0loss/FRt3/assignment_loss‚1.6556937817543271 1loss/SRqqt2/detection_loss‚2.5342237011150077 1loss/FBt2/assignment_loss‚0.30844344547937225 1loss/FBt3/assignment_loss‚0.08994140121109634 /loss/FRt2/detection_loss‚2.7406468351622904 1loss/SRqqt3/detection_loss‚0.9210437341141946 % _timestamp‚1.7885307234877827e+09 .loss/FBt2/detection_loss‚2.622608944773674 .loss/FBt4/detection_loss‚2.268879817591773£l¹ž$’‚(:‡ ° ëÔèÒßá gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.264' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NåVNr‚(:ç ° ëÔ¹¸‹ã disk.nvme1n1.in‚ 7.6796875% disk./.usageGB‚2716.4389152526855! disk.sdb.out‚1139.82470703125* proc.memory.percent‚18.214917182922363 network.sent‚ 1.2969948e+07 disk.md0p1.out‚ 1115.30078125 disk.rbd2.out‚ 1.91259765625 disk.nvme0n1.in‚0.609375 disk.md0.in‚ 199.5859375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚ 70.9609375) proc.memory.availableMB‚ 1207.66796875# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17 disk.rbd0.out‚ 1.962890625 disk.md0.out‚ 1115.30078125 cpu‚25.22698627885913 network.recv‚ 2.073627e+06 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0 disk.sda2.in‚128.625$ memory_percent‚92.62897968292236 disk.nvme1n1.out‚ 24085.59375 disk.md0p1.in‚ 199.5859375" disk.sdb2.out‚1139.82470703125 disk.sda.in‚128.625! disk.sda.out‚1139.82470703125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 96041.8359375) disk./.usagePercent‚75.95276898847588" disk.nvme0n1.out‚ 71956.2421875" disk.sda2.out‚1139.82470703125 disk.dm-0.in‚ 8.2890625¨GˆQž‘‚(:“ ¿ ëÔÉζà gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³jêA[’‚(:Ð ¿ ëÔÐÄœâ disk.dm-0.in‚ 8.2890625) proc.memory.availableMB‚ 1208.14453125# proc.memory.rssMB‚ 2984.33203125 disk.sda.out‚1141.599609375 disk.md0p1.out‚1117) disk./.usagePercent‚75.95276898847588 disk.sda2.in‚128.625 disk.sdb2.out‚1141.599609375 disk.sda.in‚128.625 disk.sdb.out‚1141.599609375 proc.cpu.threads‚17 network.sent‚ 1.3065492e+07 disk.md0.out‚1117 disk.dm-0.out‚96382.01953125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.sda1.in‚0$ memory_percent‚92.62607097625732 disk.nvme1n1.out‚ 24085.90625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0* proc.memory.percent‚18.214917182922363 network.recv‚ 2.077864e+06 disk.nvme0n1.in‚0.609375# disk.nvme0n1.out‚72296.11328125 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.91259765625% disk./.usageGB‚2716.4389152526855 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.sda1.out‚0 cpu‚25.226440934272468 disk.sda2.out‚1141.599609375 disk.md0.in‚ 199.5859375¸5À’“‚(:‡ ΠëÔÀô•à gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¸K5m”‚(:â ΠëÔÙìéà network.recv‚ 2.081737e+06 disk.rbd0.out‚ 1.962890625# disk.nvme0n1.out‚72361.18359375 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1143.38330078125 disk.sda.in‚128.625# disk.nvme1n1.out‚24086.06640625 disk.rbd2.out‚ 1.91259765625# proc.memory.rssMB‚ 2984.33203125% disk./.usageGB‚2716.4390144348145 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚0.609375" disk.sda2.out‚1143.38330078125 disk.dm-0.out‚96447.25 network.sent‚ 1.3071152e+07 disk.dm-0.in‚ 8.2890625! disk.sda.out‚1143.38330078125 disk.rbd2.in‚0$ memory_percent‚92.62533187866211) disk./.usagePercent‚75.95277176164971 disk.sda1.out‚0* proc.memory.percent‚18.214917182922363 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1143.38330078125 disk.md0p1.out‚ 1118.7265625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚128.625 cpu‚25.22594309225318 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 1118.7265625' proc.memory.availableMB‚ 1208.265625OÃm2•- _wandbruntime‚6163  _runtime‚6163òø¶¼Æ–À & lr-AdamW/pg1‚0.0005195056399935936 & lr-AdamW/pg2‚0.0005195056399935936 trainer/global_step‚5149 $ _timestamp‚1.788530780052989e+09 _runtime‚6163.589603109 _step‚420™PŒ»Ë—Å _step‚420 & lr-AdamW/pg1‚0.0005195056399935936 & lr-AdamW/pg2‚0.0005195056399935936 trainer/global_step‚5149 $ _timestamp‚1.788530780052989e+09 _runtime‚6163.589603109¤Í7T2˜- _wandbruntime‚6163  _runtime‚6163öy\ú'™ôO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5901058233856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107734746 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64551424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885307800562654e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31307523 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107734744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485028752310784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130443271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64551422 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5901058233856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485029005032960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439936027621376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6166189192192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419669842063152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5878995712202 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130469615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130443271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425548837775354 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95858336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31306914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434035879358976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439936027621376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5900148262400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478862791900672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238178015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130469615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107734744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419669825023792 _runtime‚6163.593316486 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434035897361920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439936955595776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5900148262400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚5149 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425547912760990 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204361 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434035897361920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6165960410112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439936955595776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434035879358976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425548837775354 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95858336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31306914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434035879358976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439936027621376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5900148262400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478862791900672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238178015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130469615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107734744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419669825023792 _runtime‚6163.593316486 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434035897361920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439936955595776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5900148262400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚5149 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425547912760990 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204361 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434035897361920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6165960410112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439936955595776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434035879358976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439936027621376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6166189192192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419669842063152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5878995712202 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceSta"ÝËɵtsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130469615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130443271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¥‰sí&2›- _wandbruntime‚6164  _runtime‚61640þo'œþM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31313320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485120167390208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478953017605120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130468102 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434117516447744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107754904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31312709 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130494472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440019696155136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419748859657696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130468102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130494472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5902179707392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440018768167424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434117498444800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95876196 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434117516447744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5901269722624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107754904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570]¶F 'ƒN H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130494472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5902179707392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440018768167424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434117498444800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95876196 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434117516447744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5901269722624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107754904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31313320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485120167390208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478953017605120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130468102 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434117516447744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107754904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31312709 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130494472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440019696155136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419748859657696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130468102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615¦êFù2ž- _wandbruntime‚6164  _runtime‚6164 ›žãUŸÏ 0loss/FRt3/assignment_loss‚1.3714376708556866 0loss/FBt1/assignment_loss‚0.5757526149168876 0loss/FRt4/assignment_loss‚3.4660707488656044 2loss/SRqqt1/assignment_loss‚1.6664894700050354 /loss/SRqqt1/detection_loss‚2.55680637160937 0loss/SRqqt3/detection_loss‚0.851074331884717 1loss/SRqqt4/detection_loss‚2.2859398978097096 /loss/FBt4/detection_loss‚2.6855763852596284 0loss/FBt4/assignment_loss‚0.5104098519307854 /loss/FRt2/assignment_loss‚3.536475700360757 2loss/FBt3/assignment_loss‚0.013451866490333093 /loss/FBt3/detection_loss‚0.5127015306433249 trainer/global_step‚5149 % _timestamp‚1.7885307811992989e+09 1loss/SRqqt4/assignment_loss‚1.665651284796851 1loss/FBt2/assignment_loss‚0.14546573122900305 -loss/FRt4/detection_loss‚2.22363803666085 0loss/SRqqt2/detection_loss‚2.202796433282935 1loss/SRqqt2/assignment_loss‚1.972769706307546 /loss/FRt3/detection_loss‚1.8026844128837873 _runtime‚6164.734898042 2loss/SRqqt3/assignment_loss‚0.9627348718277717 .loss/FRt1/detection_loss‚3.099727923219854 /loss/FBt1/detection_loss‚2.1294775472746954 .loss/FBt2/detection_loss‚2.355553767897866 %loss/total_loss‚44.21586678000736 epoch‚4 _step‚423 /loss/FRt2/detection_loss‚2.6696640404433074 .loss/FRt1/assignment_loss‚2.95351658355106·-×Z Ô /loss/FRt2/detection_loss‚2.6696640404433074 .loss/FRt1/assignment_loss‚2.95351658355106 0loss/FRt3/assignment_loss‚1.3714376708556866 0loss/FBt1/assignment_loss‚0.5757526149168876 0loss/FRt4/assignment_loss‚3.4660707488656044 2loss/SRqqt1/assignment_loss‚1.6664894700050354 /loss/SRqqt1/detection_loss‚2.55680637160937 0loss/SRqqt3/detection_loss‚0.851074331884717 1loss/SRqqt4/detection_loss‚2.2859398978097096 /loss/FBt4/detection_loss‚2.6855763852596284 0loss/FBt4/assignment_loss‚0.5104098519307854 /loss/FRt2/assignment_loss‚3.536475700360757 2loss/FBt3/assignment_loss‚0.013451866490333093 /loss/FBt3/detection_loss‚0.5127015306433249 trainer/global_step‚5149 % _timestamp‚1.7885307811992989e+09 1loss/SRqqt4/assignment_loss‚1.665651284796851 1loss/FBt2/assignment_loss‚0.14546573122900305 -loss/FRt4/detection_loss‚2.22363803666085 0loss/SRqqt2/detection_loss‚2.202796433282935 1loss/SRqqt2/assignment_loss‚1.972769706307546 /loss/FRt3/detection_loss‚1.8026844128837873 _runtime‚6164.734898042 2loss/SRqqt3/assignment_loss‚0.9627348718277717 .loss/FRt1/detection_loss‚3.099727923219854 /loss/FBt1/detection_loss‚2.1294775472746954 .loss/FBt2/detection_loss‚2.355553767897866 %loss/total_loss‚44.21586678000736 epoch‚4 _step‚423§Y4ê“¡‚(:ˆ Ý ëÔªÄÄß gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëa”?]¢‚(:Ò Ý ëÔæÃûà network.sent‚ 1.3088e+07 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.out‚72625.75 disk.sda.in‚128.625$ memory_percent‚92.62769222259521 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125 cpu‚25.225434734480338 network.recv‚ 2.086561e+06 disk.sda2.out‚1144.701171875 disk.sdb2.out‚1144.701171875 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 disk.md0.out‚ 1119.9765625 disk.dm-0.out‚ 96712.03125 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1144.701171875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1144.701171875 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.in‚ 199.5859375$ disk./.usageGB‚2715.908878326416 disk.rbd2.out‚ 1.91259765625 disk.dm-0.in‚ 8.2890625* proc.memory.percent‚18.214917182922363( disk./.usagePercent‚75.9379489341797 disk.nvme0n1.in‚0.609375 disk.nvme1n1.out‚ 24086.28125 disk.md0p1.out‚ 1119.9765625) proc.memory.availableMB‚ 1207.87890625!Õ^Q“£‚(:ˆ ì ëÔð¸à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.947' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.48675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s¾1Zk¤‚(:à ì ëÔ¦ÿ„á network.sent‚ 1.3185159e+07 network.recv‚ 2.091948e+06 disk.rbd0.out‚ 1.962890625 disk.sdb2.out‚1146.080078125$ disk./.usageGB‚2715.908832550049 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.91259765625" disk.nvme0n1.out‚ 72835.4140625 disk.sda2.in‚128.625# proc.memory.rssMB‚ 2984.33203125 disk.sdb1.in‚0 disk.sda2.out‚1146.080078125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.in‚ 199.5859375 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0.609375 disk.dm-0.in‚ 8.2890625 disk.sdb.out‚1146.080078125 disk.sda.out‚1146.080078125 disk.md0p1.out‚ 1121.3046875$ memory_percent‚92.62845516204834 disk.md0.out‚ 1121.3046875 disk.dm-0.out‚96921.87109375) proc.memory.availableMB‚ 1207.75390625* proc.memory.percent‚18.214917182922363 cpu‚25.224929322016624( disk./.usagePercent‚75.9379476542533 disk.sda1.in‚0# disk.nvme1n1.out‚24086.45703125%AC”¥‚(:‰ û ëÔµ«§à gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù Òn¦‚(:ã û ëÔÿŠÚà# disk.nvme1n1.out‚24086.98828125" disk.nvme0n1.out‚ 73045.9765625 disk.sdb2.out‚1147.568359375 disk.sdb1.in‚0 disk.sda1.out‚0 cpu‚25.22447003707623 disk.dm-0.out‚97132.96484375 disk.sdb.out‚1147.568359375 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.962890625) proc.memory.availableMB‚ 1207.87890625 disk.md0.out‚ 1122.73046875 disk.dm-0.in‚ 8.2890625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625* proc.memory.percent‚18.214917182922363 disk.sda2.in‚128.625 disk.md0p1.out‚ 1122.73046875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚92.62769222259521 network.recv‚ 2.095552e+06) disk./.usagePercent‚75.93872936929323% disk./.usageGB‚2715.9367904663086 disk.sda2.out‚1147.568359375 disk.sdb.in‚ 70.9609375 disk.md0.in‚ 199.5859375 disk.sda.out‚1147.568359375 disk.nvme1n1.in‚ 7.6796875 disk.rbd2.in‚0 disk.rbd2.out‚ 1.91259765625# proc.memory.rssMB‚ 2984.33203125 network.sent‚ 1.3188345e+07 disk.nvme0n1.in‚0.609375 proc.cpu.threads‚17Ãw’§‚(:‡ Š¡ëÔÂܦà gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™§ëhp¨‚(:å Š¡ëÔÜÞòà disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 disk.sda.in‚128.625! disk.sda.out‚1149.76708984375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0" disk.nvme0n1.out‚ 73162.5859375 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375# disk.nvme1n1.out‚24087.16015625$ proc.memory.availableMB‚1207.875# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.91259765625 disk.dm-0.out‚97234.87109375 cpu‚25.223968369764155 network.recv‚ 2.099373e+06" disk.sda2.out‚1149.76708984375 disk.md0.out‚ 1124.8671875 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1149.76708984375 disk.nvme1n1.in‚ 7.6796875) disk./.usagePercent‚75.93877480667992% disk./.usageGB‚2715.9384155273438 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1149.76708984375 disk.sda1.in‚0 disk.md0p1.out‚ 1124.8671875 disk.sda1.out‚0 disk.nvme0n1.in‚0.609375 disk.dm-0.in‚ 8.2890625$ memory_percent‚92.62771606445312 proc.cpu.threads‚17 network.sent‚ 1.3196216e+07* proc.memory.percent‚18.214917182922363¿Ö‡2©- _wandbruntime‚6221  _runtime‚6221¦‹²ÆªÀ & lr-AdamW/pg2‚0.0005146956147596124 trainer/global_step‚5199 % _timestamp‚1.7885308375414412e+09 _runtime‚ 6221.07710814 _step‚424 & lr-AdamW/pg1‚0.0005146956147596124b4ˆÔË«Å % _timestamp‚1.7885308375414412e+09 _runtime‚ 6221.07710814 _step‚424 & lr-AdamW/pg1‚0.0005146956147596124 & lr-AdamW/pg2‚0.0005146956147596124 trainer/global_step‚5199¨‚&}X2¬- _wandbruntime‚6221  _runtime‚62210 ý¸ ­ôO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240457263 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108742744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96751864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5956235687424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885308375426486e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5933964992306 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6224984411648 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚444073997312000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483374101063168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131714517 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65154674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 Ož3ÿB?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108742746 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5957145658880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934872967310 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108742744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108742746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65154672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489599085474816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131688173 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429555594757662 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _runtime‚6221.078333905 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444073069337600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131688173 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483374077123072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423620721790352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423620704750992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 trainer/global_step‚5199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444073997312000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438116833650176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96751251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489598849529856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438116851653120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚444073069337600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31596579 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131688173 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483374077123072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423620721790352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423620704750992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 trainer/global_step‚5199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444073997312000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438116833650176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96751251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489598849529856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438116851653120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚444073069337600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31596579 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5957145658880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934872967310 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108742744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108742746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65154672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489599085474816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131688173 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429555594757662 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _runtime‚6221.078333905 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444073069337600©†MD$2¯- _wandbruntime‚6222  _runtime‚6222=Öëî'°üM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444156737871360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108762904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438198452736000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96769012 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429635729625692 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429634804611120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5935082265584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131713004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96769659 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108762906 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483464326767616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438198452736000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489690486163968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885308386147606e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483464302827520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5935990240796 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423699739384896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚ 6222.1541851 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438198470738944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5958267132416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226159396352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108762904 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131739374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5958267132416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444156737871360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65166737 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚426 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225922238976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ±sˆh±N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438198452736000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96769012 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429635729625692 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429634804611120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5935082265584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131713004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96769659 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108762906 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483464326767616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438198452736000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489690486163968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885308386147606e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483464302827520 \JDeviceStatsMonitor.on_train_batch_end/requested_byžIÈŸtes.small_pool.allocated‚ 5935990240796 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423699739384896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚ 6222.1541851 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438198470738944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5958267132416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226159396352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108762904 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131739374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5958267132416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444156737871360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65166737 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚426 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225922238976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444156737871360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108762904ªye·Â2²- _wandbruntime‚6222  _runtime‚6222_ÚZ³Ô 2loss/SRqqt1/assignment_loss‚1.8652790788047273 0loss/FBt2/assignment_loss‚0.1515432039375858 /loss/FRt1/detection_loss‚3.2549502301216124 .loss/FRt4/detection_loss‚2.366697364087616 1loss/SRqqt3/detection_loss‚0.8560879382404308 /loss/FBt3/detection_loss‚0.6118801805262383 trainer/global_step‚5199 _runtime‚6222.155535746 1loss/SRqqt1/detection_loss‚2.4897741223203727 2loss/SRqqt3/assignment_loss‚0.9471320923354513 epoch‚4 0loss/FRt2/assignment_loss‚2.8556843803774927 0loss/FBt1/assignment_loss‚0.3519619217567036 0loss/SRqqt4/detection_loss‚2.055992578989581 .loss/FBt2/detection_loss‚2.239167121740488 _step‚427 /loss/FRt1/assignment_loss‚3.480537090301514 1loss/FBt3/assignment_loss‚0.07897428328082277 2loss/FBt4/assignment_loss‚0.021125438070157543 .loss/FBt1/detection_loss‚2.010492002262789 /loss/FBt4/detection_loss‚3.3050507392202104 0loss/FRt4/assignment_loss‚3.0223558956668493 /loss/FRt3/detection_loss‚1.4750858976816137 0loss/SRqqt2/detection_loss‚2.199844514068804 0loss/FRt3/assignment_loss‚1.2029396643241246 2loss/SRqqt2/assignment_loss‚1.5366846078908758 2loss/SRqqt4/assignment_loss‚1.6104011211642308 -loss/FRt2/detection_loss‚2.77677188958326 %loss/total_loss‚42.76641335675355 % _timestamp‚1.7885308386202133e+09+Uø°_´Ù 0loss/FBt1/assignment_loss‚0.3519619217567036 0loss/SRqqt4/detection_loss‚2.055992578989581 .loss/FBt2/detection_loss‚2.239167121740488 _step‚427 /loss/FRt1/assignment_loss‚3.480537090301514 1loss/FBt3/assignment_loss‚0.07897428328082277 2loss/FBt4/assignment_loss‚0.021125438070157543 .loss/FBt1/detection_loss‚2.010492002262789 /loss/FBt4/detection_loss‚3.3050507392202104 0loss/FRt4/assignment_loss‚3.0223558956668493 /loss/FRt3/detection_loss‚1.4750858976816137 0loss/SRqqt2/detection_loss‚2.199844514068804 0loss/FRt3/assignment_loss‚1.2029396643241246 2loss/SRqqt2/assignment_loss‚1.5366846078908758 2loss/SRqqt4/assignment_loss‚1.6104011211642308 -loss/FRt2/detection_loss‚2.77677188958326 %loss/total_loss‚42.76641335675355 % _timestamp‚1.7885308386202133e+09 2loss/SRqqt1/assignment_loss‚1.8652790788047273 0loss/FBt2/assignment_loss‚0.1515432039375858 /loss/FRt1/detection_loss‚3.2549502301216124 .loss/FRt4/detection_loss‚2.366697364087616 1loss/SRqqt3/detection_loss‚0.8560879382404308 /loss/FBt3/detection_loss‚0.6118801805262383 trainer/global_step‚5199 _runtime‚6222.155535746 1loss/SRqqt1/detection_loss‚2.4897741223203727 2loss/SRqqt3/assignment_loss‚0.9471320923354513 epoch‚4 0loss/FRt2/assignment_loss‚2.8556843803774927«ãe‡”µ‚(:‰ ™¡ëÔÕÒ„á gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>OŠýh¶‚(:Ý ™¡ëԧƞá disk.sdb2.in‚ 70.9609375 network.recv‚ 2.104166e+06) disk./.usagePercent‚75.94228159165502$ disk./.usageGB‚2716.063835144043 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 2984.33203125$ memory_percent‚92.63076782226562 cpu‚25.223507923156802! disk.sda2.out‚1151.1455078125 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 8.2890625 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.md0.out‚ 1126.203125! disk.sdb2.out‚1151.1455078125 disk.sda.out‚1151.1455078125 disk.rbd0.out‚ 1.962890625! disk.nvme0n1.out‚ 73409.859375$ proc.memory.availableMB‚1207.375 disk.sdb.out‚1151.1455078125# disk.nvme1n1.out‚24087.38671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.91259765625 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625* proc.memory.percent‚18.214917182922363 disk.rbd2.in‚0 network.sent‚ 1.3200154e+07 disk.dm-0.out‚97497.24609375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1126.203125 proc.cpu.threads‚17 disk.nvme0n1.in‚0.609375ÍG’·‚(:‡ ¨¡ëÔ‹£›à gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ŠAv¸‚(:ë ¨¡ëÔê¦ìà network.recv‚ 2.109206e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 199.5859375" disk.sdb2.out‚1152.49267578125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 1127.48828125 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sda2.in‚128.625) disk./.usagePercent‚75.94577611066892% disk./.usageGB‚2716.1888160705566 disk.dm-0.in‚ 8.2890625! disk.sda.out‚1152.49267578125 disk.md0p1.out‚ 1127.48828125# disk.nvme0n1.out‚73684.58203125 disk.sda.in‚128.625" disk.nvme1n1.out‚ 24087.6015625 disk.sdb1.out‚0" disk.sda2.out‚1152.49267578125 disk.dm-0.out‚97772.18359375 network.sent‚ 1.3309839e+07 disk.rbd2.out‚ 1.91259765625 disk.nvme0n1.in‚0.609375! disk.sdb.out‚1152.49267578125# proc.memory.rssMB‚ 2984.33203125 cpu‚25.22305580891845 disk.rbd0.out‚ 1.962890625 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.in‚ 199.5859375$ memory_percent‚92.62945652008057) proc.memory.availableMB‚ 1207.58984375* proc.memory.percent‚18.214917182922363×;(@”¹‚(:‰ ·¡ëÔ¦Âà gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.995' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.74875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ¹(ªqº‚(:æ ·¡ëÔߦõà disk.sdb1.out‚0 disk.nvme0n1.in‚0.609375 disk.sda.in‚128.625 proc.cpu.threads‚17% disk./.usageGB‚2716.1888160705566 disk.nvme0n1.out‚ 73907.03125! disk.sdb.out‚1154.43017578125! disk.sda.out‚1154.43017578125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.91259765625 disk.sdb.in‚ 70.9609375 network.recv‚ 2.113022e+06" disk.sda2.out‚1154.43017578125 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚24087.83984375& proc.memory.availableMB‚ 1207.59375* proc.memory.percent‚18.214917182922363 disk.dm-0.in‚ 8.2890625" disk.sdb2.out‚1154.43017578125 network.sent‚ 1.331313e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1129.34765625 disk.md0p1.out‚ 1129.34765625) disk./.usagePercent‚75.94577611066892 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.dm-0.out‚97994.87109375 disk.nvme1n1.in‚ 7.6796875$ memory_percent‚92.62943267822266# proc.memory.rssMB‚ 2984.33203125 cpu‚25.222601982981647'€¡ö”»‚(:‰ Æ¡ëÔ«¸³à gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë»USn¼‚(:ã Æ¡ëÔ£®–á! disk.sdb.out‚1155.88427734375 cpu‚25.222148794855077 proc.cpu.threads‚17" disk.sda2.out‚1155.88427734375 disk.dm-0.out‚98269.74609375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0# proc.memory.rssMB‚ 2984.33203125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.rbd2.out‚ 1.91259765625 disk.sda2.in‚128.625 network.sent‚ 1.3318738e+07 disk.md0.out‚ 1130.734375 disk.md0p1.out‚ 1130.734375% disk./.usageGB‚2716.4388160705566" disk.sdb2.out‚1155.88427734375! disk.sda.out‚1155.88427734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0.609375) disk./.usagePercent‚75.95276621530205# disk.nvme1n1.out‚24087.99609375 disk.rbd0.in‚0 disk.nvme0n1.out‚74181.75 disk.md0.in‚ 199.5859375$ memory_percent‚92.62945652008057) proc.memory.availableMB‚ 1207.58984375* proc.memory.percent‚18.214917182922363 disk.rbd0.out‚ 1.962890625 network.recv‚ 2.116893e+06 disk.sda.in‚128.625 disk.dm-0.in‚ 8.2890625Üó®>2½- _wandbruntime‚6278  _runtime‚6278=Õó6ǾÁ _runtime‚6278.948668652 _step‚428 & lr-AdamW/pg1‚0.0005098545806878236 & lr-AdamW/pg2‚0.0005098545806878236 trainer/global_step‚5249 % _timestamp‚1.7885308954130409e+09žøÎ]Ì¿Æ _step‚428 & lr-AdamW/pg1‚0.0005098545806878236 & lr-AdamW/pg2‚0.0005098545806878236 trainer/global_step‚5249 % _timestamp‚1.7885308954130409e+09 _runtime‚6278.948668652¬¯?=2À- _wandbruntime‚6278  _runtime‚6278 ÷6Où'ÁóO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6283433218560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433562351739970 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5990750222418 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31887271 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚5249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442197805944320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442197787941376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442197787941376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433561426725606 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚494168795564032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6012323112448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109750746 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97645193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31887880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427571584478192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚494169039897600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242710165 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109750744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97645804 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65757922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132959419 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427571601517552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448210111053824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109750746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6283653612032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132933075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448210111053824þò§ýÂøO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442197787941376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433561426725606 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚494168795564032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6012323112448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109750746 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97645193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31887880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427571584478192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚494169039897600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242710165 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109750744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97645804 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65757922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132959419 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427571601517552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448210111053824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109750746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6283653612032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132933075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚¶P€ü 609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448210111053824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6283433218560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433562351739970 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5990750222418 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31887271 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚5249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442197805944320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442197787941376­º¤+/2Ã- _wandbruntime‚6280  _runtime‚6280-MÃ'ÄüM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97663043 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6014354557440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109770906 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448292851599872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚5249 _runtime‚6280.124084378 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991867495904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442279425030144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109770906 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242728810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487975611990016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427650619112096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6013444572672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6284595951104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109770904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65769987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97663657 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 # _timestamp‚1.78853089658669e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487975588049920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5990959520692 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109770904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442279425030144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433642486608000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132957906 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242755182 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132957906 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6284826817024 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132984276 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448293779587584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6013444572672T¤ùè'ÅN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242728810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487975611990016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427650619112096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6013444572672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6284595951104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109770904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65769987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97663657 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 # _timestamp‚1.78853089658669e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487975588049920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5990959520692 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109770904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442279425030144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433642486608000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132957906 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242755182 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132957906 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6284826817024 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132984276 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448293779587584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6013444572672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97663043 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6014354557440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109770906 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448292851599872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚5249 _runtime‚6280.124084378 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991867495904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442279425030144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109770906 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856®&=€ð2Æ- _wandbruntime‚6280  _runtime‚6280ש½¬aÇÛ epoch‚4 trainer/global_step‚5249 1loss/FBt4/assignment_loss‚0.14635809084264415 /loss/FRt4/assignment_loss‚2.919002318382263 1loss/SRqqt2/detection_loss‚1.9730868804745558 1loss/SRqqt4/detection_loss‚2.0044019857519553 %loss/total_loss‚42.49745267762591 2loss/SRqqt1/assignment_loss‚1.6361075418966788 1loss/SRqqt3/detection_loss‚0.8711216818991598 /loss/FBt2/detection_loss‚2.9593551258246102 2loss/SRqqt3/assignment_loss‚1.0020685474091506 2loss/SRqqt4/assignment_loss‚1.1863100431663425 1loss/FBt2/assignment_loss‚0.12333759648986842 /loss/FRt4/detection_loss‚2.4211892638769417 /loss/FBt1/detection_loss‚2.7538272880372547 0loss/FBt3/detection_loss‚0.48594585753662484 /loss/FBt4/detection_loss‚2.3144016199641757 % _timestamp‚1.7885308965900528e+09 .loss/FRt1/detection_loss‚3.230836954712868 _step‚431 0loss/FRt3/assignment_loss‚1.4178240991186821 1loss/FBt1/assignment_loss‚0.28460861493948914 /loss/FRt3/detection_loss‚1.3930724520000748 _runtime‚6280.125385514 0loss/FRt1/assignment_loss‚3.5075294435024262 0loss/SRqqt1/detection_loss‚2.373198468376089 /loss/FRt2/detection_loss‚2.7792301589947574 0loss/FRt2/assignment_loss‚2.9739248752593994 2loss/SRqqt2/assignment_loss‚1.6465020736724865 1loss/FBt3/assignment_loss‚0.09421169549741312ò£ÚfÈà 2loss/SRqqt4/assignment_loss‚1.1863100431663425 1loss/FBt2/assignment_loss‚0.12333759648986842 /loss/FRt4/detection_loss‚2.4211892638769417 /loss/FBt1/detection_loss‚2.7538272880372547 0loss/FBt3/detection_loss‚0.48594585753662484 /loss/FBt4/detection_loss‚2.3144016199641757 % _timestamp‚1.7885308965900528e+09 .loss/FRt1/detection_loss‚3.230836954712868 _step‚431 0loss/FRt3/assignment_loss‚1.4178240991186821 1loss/FBt1/assignment_loss‚0.28460861493948914 /loss/FRt3/detection_loss‚1.3930724520000748 _runtime‚6280.125385514 0loss/FRt1/assignment_loss‚3.5075294435024262 0loss/SRqqt1/detection_loss‚2.373198468376089 /loss/FRt2/detection_loss‚2.7792301589947574 0loss/FRt2/assignment_loss‚2.9739248752593994 2loss/SRqqt2/assignment_loss‚1.6465020736724865 1loss/FBt3/assignment_loss‚0.09421169549741312 epoch‚4 trainer/global_step‚5249 1loss/FBt4/assignment_loss‚0.14635809084264415 /loss/FRt4/assignment_loss‚2.919002318382263 1loss/SRqqt2/detection_loss‚1.9730868804745558 1loss/SRqqt4/detection_loss‚2.0044019857519553 %loss/total_loss‚42.49745267762591 2loss/SRqqt1/assignment_loss‚1.6361075418966788 1loss/SRqqt3/detection_loss‚0.8711216818991598 /loss/FBt2/detection_loss‚2.9593551258246102 2loss/SRqqt3/assignment_loss‚1.0020685474091506¯î$í“É‚(:ˆ Õ¡ëÔ¯¨Ôá gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚149.442' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.3605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ííjÊ‚(:ß Õ¡ëÔÚš–â disk.dm-0.out‚98335.57421875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0% disk./.usageGB‚2716.4388160705566! disk.sda2.out‚1157.3173828125! disk.sdb2.out‚1157.3173828125 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 network.sent‚ 1.3337938e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 1.91259765625 disk.md0.in‚ 199.5859375 disk.sda.out‚1157.3173828125! disk.nvme1n1.out‚ 24088.515625) disk./.usagePercent‚75.95276621530205# proc.memory.rssMB‚ 2984.33203125 disk.dm-0.in‚ 8.2890625 disk.sdb.out‚1157.3173828125 disk.md0p1.out‚ 1132.1171875* proc.memory.percent‚18.214917182922363 cpu‚25.2216914747348 proc.cpu.threads‚17# disk.nvme0n1.out‚74247.05859375 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0% proc.memory.availableMB‚ 1207.5625 network.recv‚ 2.121503e+06 disk.md0.out‚ 1132.1171875$ memory_percent‚92.62962341308594 disk.sda1.out‚0 disk.nvme0n1.in‚0.609375 disk.sda2.in‚128.625wIgŸ“Ë‚(:ˆ ä¡ëÔ²ÍÓà gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.534' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.8835 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>jÝð^Ì‚(:Ó ä¡ëÔÿɺá network.recv‚ 2.126855e+06) disk./.usagePercent‚75.95276621530205 disk.sdb1.in‚0 disk.sdb.out‚ 1158.6796875 disk.nvme1n1.out‚ 24088.6875 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125 cpu‚25.221290215327688 disk.rbd0.in‚0% proc.memory.availableMB‚ 1207.3125 proc.cpu.threads‚17 disk.rbd2.out‚ 1.91259765625 disk.dm-0.out‚ 98675.6640625 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚ 1158.6796875* proc.memory.percent‚18.214917182922363 disk.md0.out‚ 1133.42578125 disk.dm-0.in‚ 8.2890625 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda.out‚ 1158.6796875 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.nvme0n1.in‚0.609375% disk./.usageGB‚2716.4388160705566 disk.md0.in‚ 199.5859375 disk.md0p1.out‚ 1133.42578125" disk.nvme0n1.out‚ 74586.9765625 network.sent‚ 1.3432825e+07 disk.sdb2.out‚ 1158.6796875 disk.rbd0.out‚ 1.962890625 disk.sda2.in‚128.625$ memory_percent‚92.63114929199219R(ë/“Í‚(:ˆ ó¡ëÔ©Åóà gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hJcEu΂(:ê ó¡ëÔ룎á disk.rbd0.out‚ 1.962890625" disk.sda2.out‚1160.53369140625 disk.md0.out‚ 1135.2265625 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.dm-0.in‚ 8.2890625 cpu‚25.220808918710215 disk.sda1.in‚0 disk.nvme1n1.in‚ 7.6796875 network.sent‚ 1.3440401e+07 disk.nvme0n1.in‚0.609375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1160.53369140625) proc.memory.availableMB‚ 1207.55859375* proc.memory.percent‚18.214917182922363# disk.nvme1n1.out‚24089.08203125 network.recv‚ 2.130757e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.md0.in‚ 199.5859375% disk./.usageGB‚2716.4388389587402 disk.rbd2.out‚ 1.91259765625" disk.sdb2.out‚1160.53369140625) disk./.usagePercent‚75.95276685526524 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sdb.out‚1160.53369140625$ memory_percent‚92.62964725494385# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17" disk.nvme0n1.out‚ 74957.8515625 disk.sda2.in‚128.625 disk.dm-0.out‚99046.93359375 disk.md0p1.out‚ 1135.2265625Aº“Ï‚(:ˆ ‚¢ëÔç’ÝÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è ­WqЂ(:æ ‚¢ëÔ™ª˜á disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17 disk.rbd2.in‚0 cpu‚25.22032632148158 network.sent‚ 1.3443863e+07" disk.sdb2.out‚1162.10498046875 disk.sda.in‚128.625 disk.md0p1.out‚ 1136.7265625 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 8.2890625 disk.md0.in‚ 199.5859375! disk.sdb.out‚1162.10498046875# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0.609375 disk.md0.out‚ 1136.7265625 disk.sdb.in‚ 70.9609375) disk./.usagePercent‚75.95276632196259! disk.sda.out‚1162.10498046875 disk.rbd2.out‚ 1.91259765625 disk.dm-0.out‚99310.34765625 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875" disk.sda2.out‚1162.10498046875 network.recv‚ 2.13442e+06# disk.nvme1n1.out‚24089.23828125! disk.nvme0n1.out‚ 75221.109375 disk.sda2.in‚128.625$ memory_percent‚92.62964725494385) proc.memory.availableMB‚ 1207.55859375$ disk./.usageGB‚2716.438819885254éDç´2Ñ- _wandbruntime‚6336  _runtime‚6336ü¶*¹ÄÒ¾ trainer/global_step‚5299 % _timestamp‚1.7885309533862212e+09 _runtime‚ 6336.92167699 _step‚432 % lr-AdamW/pg1‚0.000504983546642472 % lr-AdamW/pg2‚0.000504983546642472Ü‹¬ÉÓà % lr-AdamW/pg1‚0.000504983546642472 % lr-AdamW/pg2‚0.000504983546642472 trainer/global_step‚5299 % _timestamp‚1.7885309533862212e+09 _runtime‚ 6336.92167699 _step‚432°uÝLk2Ô- _wandbruntime‚6336  _runtime‚6336~¥h·TÕôO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6069320508928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244963067 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6342384823296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMoni¡ÆÛͦ%tor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32178643 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452347152770048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110758746 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚433 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚446278742232576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚446278760235520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452348080744448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66361174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6068410537472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498738793123328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437568183707914 trainer/global_step‚5299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110758746 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚437569108722278 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6046627477526 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6068410537472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452347152770048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚446278760235520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431522481244752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492396671507968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66361172 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98539815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6069320508928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32179283 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244936721 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134177977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134204321 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885309533872597e+09•¬‡ÿ'ÖùO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98539815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6069320508928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32179283 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244936721 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134177977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134204321 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885309533872597e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6069320508928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244963067 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6342384823296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32178643 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452347152770048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110758746 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚433 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚446278742232576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚446278760235520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452348080744448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66361174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6068410537472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498738793123328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437568183707914 trainer/global_step‚5299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110758746 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚437569108722278 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6046627477526 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6068410537472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452347152770048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚446278760235520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431522481244752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492396671507968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66361172 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456±Ñ¡µ32×- _wandbruntime‚6337  _runtime‚6337ñøÓ²'ØþM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110778904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66373237 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66373239 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚446360379321344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437649243590308 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6337.869222294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98557554 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452430821303808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6046836775800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134202808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069531997696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244981712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32184317 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437648318575736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110778906 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚446360379321344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6069531997696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431601481799936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6343297866752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚5299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431601498839296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98558177 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452429893316096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110778906 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6047744751012 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446360361318400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492486897212416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452430821303808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚446360379321344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437649243590308 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6337.869222294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98557554 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚452430821303808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6046836775800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134202808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069531997696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244981712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32184317 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437648318575736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110778906 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚446360379321344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6069531997696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_freÝ™%Âe‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431601481799936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6343297866752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚5299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431601498839296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98558177 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452429893316096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110778906 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6047744751012 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446360361318400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492486897212416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452430821303808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66373237 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0²SUùM2Ú- _wandbruntime‚6337  _runtime‚6337Uk¹†ZÛÔ trainer/global_step‚5299 2loss/SRqqt4/assignment_loss‚1.4178436858579517 /loss/FRt4/detection_loss‚1.9220732040703297 % _timestamp‚1.7885309543355556e+09 /loss/FRt1/assignment_loss‚2.875817563533783 0loss/FRt3/assignment_loss‚1.3339183154778602 1loss/SRqqt4/detection_loss‚1.8598388768732548 _step‚435 0loss/FRt2/assignment_loss‚3.0011883582919836 1loss/FBt2/assignment_loss‚0.27186225402607644 .loss/FRt3/detection_loss‚1.384374553266053 1loss/SRqqt3/detection_loss‚0.8586144303349225 2loss/SRqqt2/assignment_loss‚1.2250836789608002 &loss/total_loss‚39.908678629389726 1loss/FBt1/assignment_loss‚0.20530301362810455 2loss/FBt3/assignment_loss‚0.016926493456772277 /loss/FRt1/detection_loss‚3.1036172890663147 -loss/FRt2/detection_loss‚2.45502138941902 _runtime‚6337.870829401 2loss/SRqqt1/assignment_loss‚1.4613155752420426 0loss/SRqqt1/detection_loss‚2.402440276145935 /loss/FBt2/detection_loss‚1.9979029679298401 .loss/FBt3/detection_loss‚0.785526053669552 .loss/FBt4/detection_loss‚2.330200063554864 0loss/FRt4/assignment_loss‚3.3305989450878566 /loss/FBt1/detection_loss‚2.1024506423208447 epoch‚4 0loss/SRqqt3/assignment_loss‚1.00829614645189 0loss/FBt4/assignment_loss‚0.2768894709678486 0loss/SRqqt2/detection_loss‚2.281575381755829’ =_ÜÙ .loss/FRt3/detection_loss‚1.384374553266053 1loss/SRqqt3/detection_loss‚0.8586144303349225 2loss/SRqqt2/assignment_loss‚1.2250836789608002 &loss/total_loss‚39.908678629389726 1loss/FBt1/assignment_loss‚0.20530301362810455 2loss/FBt3/assignment_loss‚0.016926493456772277 /loss/FRt1/detection_loss‚3.1036172890663147 -loss/FRt2/detection_loss‚2.45502138941902 _runtime‚6337.870829401 2loss/SRqqt1/assignment_loss‚1.4613155752420426 0loss/SRqqt1/detection_loss‚2.402440276145935 /loss/FBt2/detection_loss‚1.9979029679298401 .loss/FBt3/detection_loss‚0.785526053669552 .loss/FBt4/detection_loss‚2.330200063554864 0loss/FRt4/assignment_loss‚3.3305989450878566 /loss/FBt1/detection_loss‚2.1024506423208447 epoch‚4 0loss/SRqqt3/assignment_loss‚1.00829614645189 0loss/FBt4/assignment_loss‚0.2768894709678486 0loss/SRqqt2/detection_loss‚2.281575381755829 trainer/global_step‚5299 2loss/SRqqt4/assignment_loss‚1.4178436858579517 /loss/FRt4/detection_loss‚1.9220732040703297 % _timestamp‚1.7885309543355556e+09 /loss/FRt1/assignment_loss‚2.875817563533783 0loss/FRt3/assignment_loss‚1.3339183154778602 1loss/SRqqt4/detection_loss‚1.8598388768732548 _step‚435 0loss/FRt2/assignment_loss‚3.0011883582919836 1loss/FBt2/assignment_loss‚0.27186225402607644³Ö’þ’Ý‚(:‡ ‘¢ëÔñû†à gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¬kÞ‚(:à ‘¢ëÔîë¦â disk.sdb2.out‚ 1163.47265625 disk.sda.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.962890625) proc.memory.availableMB‚ 1207.55859375 proc.cpu.threads‚17 network.recv‚ 2.139084e+06 disk.sda2.out‚ 1163.47265625 disk.dm-0.out‚ 99535.9296875 disk.sdb.out‚ 1163.47265625# disk.nvme1n1.out‚24089.56640625 disk.sdb1.in‚0 disk.sda1.out‚0 cpu‚25.219876675459144 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 8.2890625 disk.sdb.in‚ 70.9609375 disk.rbd2.in‚0$ memory_percent‚92.62964725494385 disk.md0p1.out‚ 1138.03515625 disk.sda1.in‚0) disk./.usagePercent‚75.93794306785043% disk./.usageGB‚2715.9086685180664 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚ 1.91259765625# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363 disk.nvme0n1.in‚0.609375 disk.md0.out‚ 1138.03515625 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0# disk.nvme0n1.out‚75446.36328125 disk.sda.out‚ 1163.47265625 disk.sdb1.out‚0 network.sent‚ 1.3462894e+07K|Mw’ß‚(:‡  ¢ëÔ膱à gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q¤tÚnà‚(:ã  ¢ëÔï£ñà disk.rbd0.in‚0 network.recv‚ 2.144341e+06) disk./.usagePercent‚75.93794402779523" disk.sda2.out‚1165.34326171875 proc.cpu.threads‚17 disk.rbd0.out‚ 1.962890625 disk.rbd2.out‚ 1.97509765625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 75466.1171875 disk.dm-0.out‚ 99551.6015625" disk.sdb2.out‚1165.34326171875! disk.sda.out‚1165.34326171875 disk.sdb1.in‚0! disk.nvme1n1.out‚ 24089.734375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.out‚0% proc.memory.availableMB‚ 1207.3125$ disk./.usageGB‚2715.908702850342 disk.nvme0n1.in‚0.609375 disk.sda.in‚128.625$ memory_percent‚92.63114929199219* proc.memory.percent‚18.214917182922363 cpu‚25.21948930054193 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.dm-0.in‚ 8.2890625 disk.md0p1.out‚ 1139.83984375 network.sent‚ 1.3557732e+07 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1139.83984375! disk.sdb.out‚1165.34326171875# proc.memory.rssMB‚ 2984.33203125©zQ“á‚(:ˆ ¯¢ëÔ­£ºà gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÐÚlâ‚(:á ¯¢ëÔš…÷à! disk.sda.out‚1166.78662109375 network.sent‚ 1.356305e+07 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0% disk./.usageGB‚2715.8520584106445 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚75761.65234375( proc.memory.availableMB‚ 1207.5546875 proc.cpu.threads‚17 network.recv‚ 2.14802e+06! disk.sdb.out‚1166.78662109375 disk.sda.in‚128.625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1141.21875 disk.dm-0.in‚ 8.2890625 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1166.78662109375# disk.nvme1n1.out‚24090.05078125 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1141.21875" disk.sda2.out‚1166.78662109375 disk.rbd0.out‚ 1.962890625$ memory_percent‚92.62967109680176 cpu‚25.21905252087195) disk./.usagePercent‚75.93636022555374 disk.dm-0.out‚ 99851.703125 disk.rbd0.in‚0 disk.rbd2.out‚ 1.97509765625 disk.nvme0n1.in‚0.609375# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363 W} žã‚(:“ ¾¢ëÔÁÕÔà gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤³±Êlä‚(:á ¾¢ëÔƒþ»á disk.nvme0n1.in‚0.609375 disk.sda2.in‚128.625! disk.sdb2.out‚1168.3837890625) proc.memory.availableMB‚ 1207.55078125# disk.nvme1n1.out‚24176.46484375 disk.rbd0.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.214917182922363) disk./.usagePercent‚75.93871358353448 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.out‚ 1142.765625 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.dm-0.in‚ 8.2890625 disk.sda.out‚1168.3837890625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 100095.359375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 cpu‚25.218613807561166 network.recv‚ 2.1516e+06 disk.sda.in‚128.625# disk.nvme0n1.out‚75918.89453125! disk.sda2.out‚1168.3837890625 disk.md0.out‚ 1142.765625# proc.memory.rssMB‚ 2984.33203125 disk.sdb.out‚1168.3837890625% disk./.usageGB‚2715.9362258911133 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.rbd2.out‚ 1.9990234375 disk.md0.in‚ 199.5859375$ memory_percent‚92.62969493865967 proc.cpu.threads‚17 network.sent‚ 1.3570993e+07g¼‰2å- _wandbruntime‚6394  _runtime‚6394Ø zÇæÁ & lr-AdamW/pg1‚0.0005000835277397518 & lr-AdamW/pg2‚0.0005000835277397518 trainer/global_step‚5349 % _timestamp‚1.7885310109522767e+09 _runtime‚6394.487971711 _step‚436\ŽS¯ÌçÆ trainer/global_step‚5349 % _timestamp‚1.7885310109522767e+09 _runtime‚6394.487971711 _step‚436 & lr-AdamW/pg1‚0.0005000835277397518 & lr-AdamW/pg2‚0.0005000835277397518´“ð÷2è- _wandbruntime‚6394  _runtime‚6394+8ñ§ù'éóO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66964424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32471751 trainer/global_step‚5349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6125407933952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111766746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503308832658944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496907932790272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111766744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6400899868672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6124497962496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450359714526720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435473360971952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6101596757630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450359696523776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503309072798208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 % _timestamp‚1.7885310109538598e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6125407933952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚456485122460672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135422879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441574940690222 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456485122460672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450359714526720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 6394.48977674 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32471127 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441575865704586 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456484194486272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6401116067840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99435549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456484194486272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66964422 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6124497962496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268Ðün±šêøO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135449223 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99436175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚435473343932592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135422879 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450359696523776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111766746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496907956730368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247189623 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456484194486272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6401116067840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99435549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456484194486272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66964422 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6124497962496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66964424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32471751 trainer/global_step‚5349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6125407933952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.alï'ròdlocated‚ 111766746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503308832658944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496907932790272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111766744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6400899868672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6124497962496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450359714526720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435473360971952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6101596757630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450359696523776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503309072798208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 % _timestamp‚1.7885310109538598e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6125407933952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚456485122460672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135422879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441574940690222 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456485122460672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450359714526720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 6394.48977674 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32471127 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441575865704586 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456567863020032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99453954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435552361527136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496998158494720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435552378566496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6402071450624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6102714030908 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111786904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135447710 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6126529407488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450441315609600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6402296025088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6125619422720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111786904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135447710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450441333612544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450441333612544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32476835 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99453322 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503400478459904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚5349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503400229945344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456567863020032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32477465 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496998182434816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450441315609600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66976487 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 $ _timestamp‚1.788531012126131e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6125619422720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144¢N 'í‚N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99453322 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503400478459904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚5349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503400229945344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456567863020032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32477465 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496998182434816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450441315609600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66976487 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 $ _timestamp‚1.788531012126131e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6125619422720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111786906 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441655075558044 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456567863020032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99453954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435552361527136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496998158494720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435552378566496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6402071450624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6102714030908 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111786904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135447710 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6126529407488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450441315609600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6402296025088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6125619422720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111786904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135447710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450441333612544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450441333612544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32476835 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501419218012672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439424223659792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460622164176896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460621236202496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454440668817920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460622164176896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249442525 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460621236202496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6157474012738 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100329563 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454440668817920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445582622686894 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136694125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6459768217600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32761889 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112774746 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445581697672530 $ _timestamp‚1.788531067540348e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501419218012672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439424223659792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460622164176896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460621236202496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454440668817920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460622164176896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249442525 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460621236202496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6157474012738 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100329563 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454440668817920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445582622686894 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136694125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6459768217600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32761889 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112774746 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445581697672530 $ _timestamp‚1.788531067540348e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439503241254336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6159499261228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507970147740672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460704904736256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454522287903744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460704904736256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100346727 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67579739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249487516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112794904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136692612 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67579737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445661832540352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136718982 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6182616832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454522287903744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112794904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112794906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6182616832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181706847744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚5399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460703976748544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚6452.126478511 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6158591286016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885310685902622e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136692612 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112794906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6460926500864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚442 HÑž…N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67579737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445661832540352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136718982 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6182616832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_endÈ#½m#/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454522287903744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112794904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112794906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6182616832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181706847744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚5399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460703976748544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚6452.126478511 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6158591286016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885310685902622e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136692612 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112794906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6460926500864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚442 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454522269900800 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439503241254336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6159499261228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507970147740672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460704904736256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454522287903744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460704904736256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100346727 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67579739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249487516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112794904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136692612 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ºÎÝmÝ2‚- _wandbruntime‚6452  _runtime‚6452oùæa]ƒ× epoch‚4 2loss/FBt2/assignment_loss‚0.026582571158462322 /loss/FRt3/detection_loss‚1.4927967522366374 1loss/SRqqt3/detection_loss‚0.9136338775329615 /loss/FBt1/detection_loss‚2.7266796231269836 /loss/FBt2/detection_loss‚1.9285883353306696 0loss/FRt2/assignment_loss‚3.0889496460105432 /loss/FRt4/assignment_loss‚2.753481166307316 2loss/SRqqt2/assignment_loss‚1.3466325034110231 1loss/SRqqt4/assignment_loss‚1.188835446536541 trainer/global_step‚5399 % _timestamp‚1.7885310685922108e+09 _runtime‚6452.127512316 2loss/FBt4/assignment_loss‚0.037366961426111324 1loss/SRqqt2/detection_loss‚1.8512394690797442 0loss/SRqqt4/detection_loss‚2.185094574519566 /loss/FBt3/detection_loss‚0.6476240735238706 &loss/total_loss‚41.714543477034596 0loss/FRt1/assignment_loss‚3.2374496336640983 /loss/FRt3/assignment_loss‚1.066949161828733 2loss/SRqqt3/assignment_loss‚1.0325033761075009 .loss/FBt4/detection_loss‚4.196946978569031 _step‚443 1loss/FBt3/assignment_loss‚0.04324888608438951 .loss/FRt4/detection_loss‚2.220666462598845 2loss/SRqqt1/assignment_loss‚1.4875407530104412 .loss/FRt1/detection_loss‚3.016456133332746 /loss/FRt2/detection_loss‚2.6233267057350806 0loss/SRqqt1/detection_loss‚2.266879093997619 0loss/FBt1/assignment_loss‚0.3350712919056807Ý*[öb„Ü 0loss/FRt1/assignment_loss‚3.2374496336640983 /loss/FRt3/assignment_loss‚1.066949161828733 2loss/SRqqt3/assignment_loss‚1.0325033761075009 .loss/FBt4/detection_loss‚4.196946978569031 _step‚443 1loss/FBt3/assignment_loss‚0.04324888608438951 .loss/FRt4/detection_loss‚2.220666462598845 2loss/SRqqt1/assignment_loss‚1.4875407530104412 .loss/FRt1/detection_loss‚3.016456133332746 /loss/FRt2/detection_loss‚2.6233267057350806 0loss/SRqqt1/detection_loss‚2.266879093997619 0loss/FBt1/assignment_loss‚0.3350712919056807 epoch‚4 2loss/FBt2/assignment_loss‚0.026582571158462322 /loss/FRt3/detection_loss‚1.4927967522366374 1loss/SRqqt3/detection_loss‚0.9136338775329615 /loss/FBt1/detection_loss‚2.7266796231269836 /loss/FBt2/detection_loss‚1.9285883353306696 0loss/FRt2/assignment_loss‚3.0889496460105432 /loss/FRt4/assignment_loss‚2.753481166307316 2loss/SRqqt2/assignment_loss‚1.3466325034110231 1loss/SRqqt4/assignment_loss‚1.188835446536541 trainer/global_step‚5399 % _timestamp‚1.7885310685922108e+09 _runtime‚6452.127512316 2loss/FBt4/assignment_loss‚0.037366961426111324 1loss/SRqqt2/detection_loss‚1.8512394690797442 0loss/SRqqt4/detection_loss‚2.185094574519566 /loss/FBt3/detection_loss‚0.6476240735238706 &loss/total_loss‚41.714543477034596»¨¦¹+’…‚(:‡ ‰£ëԽ׫ç gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|]k†‚(:à ‰£ëÔ›ëùê* proc.memory.percent‚18.214917182922363 cpu‚25.21620274376252 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0" disk.sda2.out‚1176.30419921875 disk.sdb.in‚ 70.9609375) proc.memory.availableMB‚ 1207.26953125 network.sent‚ 1.3699122e+07 network.recv‚ 2.173651e+06) disk./.usagePercent‚75.95276600198099 disk.sda1.out‚0 disk.rbd2.out‚2.125" disk.sdb2.out‚1176.30419921875 disk.md0p1.out‚ 1150.43359375 proc.cpu.threads‚17 disk.md0.out‚ 1150.43359375! disk.sdb.out‚1176.30419921875! disk.sda.out‚1176.30419921875 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 1.962890625 disk.md0.in‚ 199.5859375 disk.nvme1n1.out‚ 24517.46875$ disk./.usageGB‚2716.438808441162# disk.nvme0n1.out‚76548.33984375 disk.dm-0.in‚ 8.2890625# proc.memory.rssMB‚ 2984.33203125 disk.nvme0n1.in‚0.609375 disk.rbd0.in‚0! disk.dm-0.out‚101065.80859375 disk.sda.in‚128.625# memory_percent‚92.6314115524292ÀçQ|’‡‚(:‡ ˜£ëÔÆúöÞ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙJ„nˆ‚(:ã ˜£ëÔ±Êïß disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.nvme0n1.in‚0.609375 disk.sdb.in‚ 70.9609375$ memory_percent‚92.63148307800293 network.sent‚ 1.3806645e+07 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.out‚ 1151.81640625 disk.sda1.in‚0! disk.dm-0.out‚101394.27734375( proc.memory.availableMB‚ 1207.2578125 disk.rbd0.out‚ 1.962890625# proc.memory.rssMB‚ 2984.33203125 cpu‚25.215716895708923) disk./.usagePercent‚75.95276600198099" disk.nvme1n1.out‚ 24636.2890625 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.dm-0.in‚ 8.2890625 disk.sda1.out‚0 disk.md0.out‚ 1151.81640625" disk.sdb2.out‚1177.73681640625 disk.rbd2.out‚2.125# disk.nvme0n1.out‚76757.98828125* proc.memory.percent‚18.214917182922363 network.recv‚ 2.179223e+06$ disk./.usageGB‚2716.438808441162" disk.sda2.out‚1177.73681640625! disk.sdb.out‚1177.73681640625 disk.sda.in‚128.625! disk.sda.out‚1177.73681640625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375ì©ï”‰‚(:‰ §£ëÔª´•à gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.137' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.53425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>hedŠ‚(:Ù §£ëÔÝ‘×à disk.sda.in‚128.625 cpu‚25.215290685389025 disk.nvme1n1.in‚ 7.6796875 disk.sda1.in‚0# proc.memory.rssMB‚ 2984.33203125 network.sent‚ 1.3812384e+07) disk./.usagePercent‚75.95276600198099 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 8.2890625$ memory_percent‚92.63143539428711* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17 disk.rbd0.out‚ 1.962890625 disk.nvme0n1.in‚0.609375! disk.dm-0.out‚101700.26953125 network.recv‚ 2.18337e+06 disk.md0p1.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1179.2451171875' proc.memory.availableMB‚ 1207.265625# disk.nvme1n1.out‚24732.57421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚2.125" disk.nvme0n1.out‚ 76967.6953125 disk.md0.out‚ 1153.265625 disk.sdb.in‚ 70.9609375 disk.sda.out‚1179.2451171875 disk.md0p1.out‚ 1153.265625 disk.sda1.out‚0! disk.sda2.out‚1179.2451171875 disk.sdb.out‚1179.2451171875$ disk./.usageGB‚2716.438808441162 disk.rbd0.in‚0 disk.rbd2.in‚0dÑLj2‹- _wandbruntime‚6505  _runtime‚6505î[ÆÇŒÁ % _timestamp‚1.7885311222982469e+09 _runtime‚6505.833798694 _step‚444 & lr-AdamW/pg1‚0.0004902006258161787 & lr-AdamW/pg2‚0.0004902006258161787 trainer/global_step‚5449À¹í´ÌÆ _runtime‚6505.833798694 _step‚444 & lr-AdamW/pg1‚0.0004902006258161787 & lr-AdamW/pg2‚0.0004902006258161787 trainer/global_step‚5449 % _timestamp‚1.7885311222982469e+09¼øHçµ2Ž- _wandbruntime‚6505  _runtime‚6505yÈé"û'õO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464758277918720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458521605106176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113782746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68170924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68170922 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137939027 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6214259242850 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6237582784000 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113782746 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6236672812544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137912683 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137912683 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6237582784000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚6505.834954749 _step‚445 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6518495822848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113782744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33051671 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512449022998016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458521623109120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458521605106176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512448776567296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464759205893120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449588454654838 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6213351267846 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33052265 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113782744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 trainer/global_step‚5449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464759205893120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458521623109120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930527175168c·~æKúO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251721773 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443375103386992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464758277918720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137939027 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6236672812544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458521623109120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930527175168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464758277918720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458521605106176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113782746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68170924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68170922 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137939027 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6214259242850 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6237582784000 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113782746 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6236672812544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137912683 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137912683 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6237582784000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚6505.834954749 _step‚445 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6518495822848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113782744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚330a×ôTµ51671 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512449022998016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458521623109120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458521605106176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512448776567296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464759205893120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449588454654838 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6213351267846 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33052265 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113782744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 trainer/global_step‚5449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464759205893120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101241028 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6215376516336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68182987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251740418 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113802906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚506020728939520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603242194944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512540431818752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464841946452480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449668589522660 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚6506.875193613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464841946452480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449669514537232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464841018464768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101240420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6519469044736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113802904 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6519678939136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443454120981536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603224192000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113802904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603224192000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113802906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6238704257536 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6237794272768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603242194944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33057433 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443454138020896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512540197984256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506020752879616 _step‚446 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137937514 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6214468541124 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6238704257536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113802906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚506020728939520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603242194944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512540431818752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464841946452480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449668589522660 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚6506.875193613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464841946452480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449669514537232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464841018464768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101240420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6519469044736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113802904 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6519678939136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443454120981536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603224192000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113802904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603224192000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113802906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6238704257536 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6237794272768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603242194944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33057433 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443454138020896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512540197984256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506020752879616 _step‚446 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137937514 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6214468541124 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6238704257536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101241028 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6215376516336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68182987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784¾Œ½Þ2”- _wandbruntime‚6506  _runtime‚6506|o".Z•Ô 2loss/SRqqt4/assignment_loss‚1.9154946388676763 /loss/FRt3/detection_loss‚1.5683305222254533 .loss/FBt1/detection_loss‚2.350267171859741 .loss/FRt2/detection_loss‚2.512268291415871 2loss/SRqqt1/assignment_loss‚1.4404260189582905 2loss/SRqqt2/assignment_loss‚1.3589329038831321 0loss/FBt2/assignment_loss‚0.3423070503174088 0loss/SRqqt4/detection_loss‚2.433052712678909 .loss/FBt2/detection_loss‚2.215368619986943 epoch‚4 _runtime‚6506.876185698 _step‚447 /loss/FBt3/detection_loss‚0.5944849727127481 /loss/FBt4/detection_loss‚1.6265218822579635 trainer/global_step‚5449 0loss/FRt1/assignment_loss‚3.6369216153115937 0loss/SRqqt1/detection_loss‚2.296327053507169 0loss/SRqqt2/detection_loss‚2.249682128429413 0loss/FRt2/assignment_loss‚3.1897256306239536 0loss/FRt3/assignment_loss‚1.1278901191858144 1loss/FBt4/assignment_loss‚0.16730891742755424 %loss/total_loss‚42.18746662784134 % _timestamp‚1.7885311233409164e+09 0loss/FRt4/assignment_loss‚3.5828160136938094 1loss/SRqqt3/detection_loss‚0.8950215393658911 .loss/FRt1/detection_loss‚3.079417756109527 2loss/SRqqt3/assignment_loss‚0.9880169464072378 1loss/FBt1/assignment_loss‚0.15499792966368528 0loss/FBt3/assignment_loss‚0.1756675141865492 /loss/FRt4/detection_loss‚2.2862186787649987^(¢_–Ù 2loss/SRqqt4/assignment_loss‚1.9154946388676763 /loss/FRt3/detection_loss‚1.5683305222254533 .loss/FBt1/detection_loss‚2.350267171859741 .loss/FRt2/detection_loss‚2.512268291415871 2loss/SRqqt1/assignment_loss‚1.4404260189582905 2loss/SRqqt2/assignment_loss‚1.3589329038831321 0loss/FBt2/assignment_loss‚0.3423070503174088 0loss/SRqqt4/detection_loss‚2.433052712678909 .loss/FBt2/detection_loss‚2.215368619986943 epoch‚4 _runtime‚6506.876185698 _step‚447 /loss/FBt3/detection_loss‚0.5944849727127481 /loss/FBt4/detection_loss‚1.6265218822579635 trainer/global_step‚5449 0loss/FRt1/assignment_loss‚3.6369216153115937 0loss/SRqqt1/detection_loss‚2.296327053507169 0loss/SRqqt2/detection_loss‚2.249682128429413 0loss/FRt2/assignment_loss‚3.1897256306239536 0loss/FRt3/assignment_loss‚1.1278901191858144 1loss/FBt4/assignment_loss‚0.16730891742755424 %loss/total_loss‚42.18746662784134 % _timestamp‚1.7885311233409164e+09 0loss/FRt4/assignment_loss‚3.5828160136938094 1loss/SRqqt3/detection_loss‚0.8950215393658911 .loss/FRt1/detection_loss‚3.079417756109527 2loss/SRqqt3/assignment_loss‚0.9880169464072378 1loss/FBt1/assignment_loss‚0.15499792966368528 0loss/FBt3/assignment_loss‚0.1756675141865492 /loss/FRt4/detection_loss‚2.2862186787649987¿|6DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68774172 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447326000153552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102116557 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6577238741504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462602577400320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462602559397376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _runtime‚ 6559.92225032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139183929 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6292760237568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510441812397568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139183929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447325983114192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468896247609344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468895319634944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102115928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885311763865132e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114790746 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468896247609344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6269228522954 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114790746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68774174 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6270136497958 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6577016250880 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6293670209024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517018804708352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462602559397376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114790744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462602577400320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517019051139072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139157585 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510441788457472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453596136651510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114790746 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468896247609344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6269228522954 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114790746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68774174 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6270136497958 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6577016250880 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6293670209024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517018804708352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462602559397376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114790744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462602577400320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517019051139072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139157585 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510441788457472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453596136651510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68774172 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447326000153552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102116557 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6577238741504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462602577400320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462602559397376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _runtime‚ 6559.92225032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139183929 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6292760237568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510441812397568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139183929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447325983114192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468896247609344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468895319634944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102115928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÁùjO2¥- _wandbruntime‚6560  _runtime‚6560— °´'¦€N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6293881697792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517110451849728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33348172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6578413747712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139182416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6294791682560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253993320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462684178483200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114810906 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114810904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453676271519540 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114810906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6271253771444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33347522 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462684196486144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6578187076096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462684196486144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139208786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68786239 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510532038102016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚6560.988574903 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468978988168704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468978060180992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253993320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114810904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447405017748096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139208786 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254019692 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102133759 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6293881697792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517110201238016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚450Á¢ØV§…N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453676271519540 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JñÆVµ$BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114810906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6271253771444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33347522 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462684196486144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6578187076096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462684196486144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139208786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68786239 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510532038102016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚6560.988574903 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468978988168704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468978060180992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253993320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114810904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447405017748096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139208786 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254019692 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102133759 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6293881697792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517110201238016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6293881697792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517110451849728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33348172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6578413747712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139182416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6294791682560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253993320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462684178483200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114810906 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114810904Â{ž\12¨- _wandbruntime‚6560  _runtime‚6560êIvÅ[©Õ /loss/FRt2/assignment_loss‚3.323243709405263 /loss/FBt3/detection_loss‚0.5694464051429137 epoch‚4 1loss/SRqqt2/assignment_loss‚1.412688044791526 2loss/SRqqt4/assignment_loss‚2.0345168069769173 1loss/FBt2/assignment_loss‚0.19495294365009938 .loss/FRt1/detection_loss‚2.907597987940817 1loss/SRqqt4/detection_loss‚2.4331700389201822 /loss/FBt2/detection_loss‚2.6528224415249295 2loss/SRqqt1/assignment_loss‚2.1222241944991627 3loss/FBt3/assignment_loss‚0.0077899305196997785 1loss/FBt4/assignment_loss‚0.21110572164191885 /loss/FRt3/detection_loss‚1.5449615107341246 # _timestamp‚1.78853117745431e+09 _step‚451 0loss/FRt1/assignment_loss‚2.9804641514113457 0loss/FRt3/assignment_loss‚1.4406897815791044 /loss/FRt4/assignment_loss‚3.107806067077481 /loss/FRt2/detection_loss‚2.3029635379226723 trainer/global_step‚5499 /loss/FRt4/detection_loss‚2.1612910237543437 0loss/SRqqt1/detection_loss‚2.359040977099003 0loss/SRqqt2/detection_loss‚1.948947804405334 .loss/FBt1/detection_loss‚2.474574676156044 %loss/total_loss‚43.39051116420913 _runtime‚6560.989613908 2loss/SRqqt3/assignment_loss‚0.9974878246404146 0loss/FBt1/assignment_loss‚0.2779875888765673 1loss/SRqqt3/detection_loss‚0.8445828712723228 /loss/FBt4/detection_loss‚3.08015512426694250òR`ªÚ /loss/FRt2/detection_loss‚2.3029635379226723 trainer/global_step‚5499 /loss/FRt4/detection_loss‚2.1612910237543437 0loss/SRqqt1/detection_loss‚2.359040977099003 0loss/SRqqt2/detection_loss‚1.948947804405334 .loss/FBt1/detection_loss‚2.474574676156044 %loss/total_loss‚43.39051116420913 _runtime‚6560.989613908 2loss/SRqqt3/assignment_loss‚0.9974878246404146 0loss/FBt1/assignment_loss‚0.2779875888765673 1loss/SRqqt3/detection_loss‚0.8445828712723228 /loss/FBt4/detection_loss‚3.0801551242669425 /loss/FRt2/assignment_loss‚3.323243709405263 /loss/FBt3/detection_loss‚0.5694464051429137 epoch‚4 1loss/SRqqt2/assignment_loss‚1.412688044791526 2loss/SRqqt4/assignment_loss‚2.0345168069769173 1loss/FBt2/assignment_loss‚0.19495294365009938 .loss/FRt1/detection_loss‚2.907597987940817 1loss/SRqqt4/detection_loss‚2.4331700389201822 /loss/FBt2/detection_loss‚2.6528224415249295 2loss/SRqqt1/assignment_loss‚2.1222241944991627 3loss/FBt3/assignment_loss‚0.0077899305196997785 1loss/FBt4/assignment_loss‚0.21110572164191885 /loss/FRt3/detection_loss‚1.5449615107341246 # _timestamp‚1.78853117745431e+09 _step‚451 0loss/FRt1/assignment_loss‚2.9804641514113457 0loss/FRt3/assignment_loss‚1.4406897815791044 /loss/FRt4/assignment_loss‚3.107806067077481àUl’«‚(:‡ ò£ëÔÁ„Æà gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.552' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ˆþg¬‚(:Ü ò£ëÔó¶§á$ disk./.usageGB‚2715.939628601074 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.214917182922363 network.sent‚ 1.3943549e+07" disk.sdb2.out‚1187.93115234375# disk.nvme1n1.out‚24875.45703125 disk.sdb1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚77738.40234375$ memory_percent‚92.63520240783691 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.out‚ 1.962890625 disk.sda2.in‚128.625" disk.sda2.out‚1187.93115234375 disk.dm-0.out‚ 102613.859375 disk.md0.out‚ 1161.65625( proc.memory.availableMB‚ 1206.6484375) disk./.usagePercent‚75.93880872472913 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.rbd2.out‚2.125 disk.dm-0.in‚ 9.1640625# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1187.93115234375 disk.md0p1.out‚ 1161.65625 disk.nvme0n1.in‚1.484375 network.recv‚ 2.205874e+06 disk.sdb2.in‚ 70.9609375 disk.md0.in‚ 199.5859375 cpu‚25.213254042995594! disk.sda.out‚1187.93115234375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 Ù$›”­‚(:‰ ¤ëÔ×ÄòÞ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]SÄi®‚(:Þ ¤ëÔþœåß disk.rbd2.in‚0* proc.memory.percent‚18.214917182922363 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.rbd2.out‚2.125$ memory_percent‚92.63303279876709) proc.memory.availableMB‚ 1207.00390625% disk./.usageGB‚2716.0648155212402 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0# proc.memory.rssMB‚ 2984.33203125 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb.out‚1189.4638671875 disk.md0p1.out‚ 1163.1328125 disk.sdb1.in‚0 cpu‚25.212819915244417 proc.cpu.threads‚17 network.sent‚ 1.4051295e+07 network.recv‚ 2.21163e+06" disk.nvme0n1.out‚ 77980.7109375 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625! disk.sda2.out‚1189.4638671875 disk.md0.out‚ 1163.1328125! disk.dm-0.out‚102856.44921875# disk.nvme1n1.out‚24875.73828125 disk.nvme0n1.in‚1.484375 disk.dm-0.in‚ 9.1640625 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚1189.4638671875 disk.sda.out‚1189.4638671875) disk./.usagePercent‚75.94230900341176>×~¶M¯‚(: ¤ëÔ™„¬ã% disk./.usageGB‚2716.1895484924316 disk.md0.out‚ 1164.609375 disk.rbd2.out‚2.125# proc.memory.rssMB‚ 2984.33203125 disk.sda2.out‚ 1191.015625 disk.sdb.out‚ 1191.015625 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1164.609375 disk.nvme0n1.in‚1.484375 cpu‚25.212423184408387 disk.sda.in‚128.625 proc.memory.availableMB‚1207* proc.memory.percent‚18.214917182922363 disk.sda2.in‚128.625 disk.rbd0.out‚ 1.962890625) disk./.usagePercent‚75.94579658949108 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚78235.73828125" memory_percent‚92.633056640625 disk.dm-0.out‚103111.6328125 disk.sdb2.out‚ 1191.015625 disk.sda.out‚ 1191.015625# disk.nvme1n1.out‚24875.89453125 disk.sdb1.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.4057432e+07 network.recv‚ 2.216121e+06 disk.nvme1n1.in‚ 7.6796875 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 9.1640625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0Oµzy”°‚(:‰ ¤ëÔšÿÀö gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.161' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.29025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýd¬2±- _wandbruntime‚6613  _runtime‚661314DzÁ & lr-AdamW/pg1‚0.0004802141128156827 & lr-AdamW/pg2‚0.0004802141128156827 trainer/global_step‚5549 % _timestamp‚1.7885312303511138e+09 _runtime‚6613.886762732 _step‚4525-ypÌ³Æ trainer/global_step‚5549 % _timestamp‚1.7885312303511138e+09 _runtime‚6613.886762732 _step‚452 & lr-AdamW/pg1‚0.0004802141128156827 & lr-AdamW/pg2‚0.0004802141128156827Äa†Ïs2´- _wandbruntime‚6613  _runtime‚6613½ìUû'µõO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69377424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6325105778062 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6349757634048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚514953097619968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6635972530688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885312303523407e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140428831 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457602893633818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473033289325568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚466683531691520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521588819525632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140428831 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103011670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326013753066 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140402487 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚466683513688576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115798744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473032361351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473032361351168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚ 6613.88841431 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466683531691520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473033289325568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69377422 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521589070150656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚5549 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115798746 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457601968619454 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256227577 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140402487 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33633620 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514953073679872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115798744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6635745845760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451276862841392‡o %¶úO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140428831 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103011670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326013753066 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140402487 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚466683513688576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115798744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473032361351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473032361351168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚ 6613.88841431 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466683531691520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473033289325568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69377422 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521589070150656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚5549 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115798746 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457601968619454 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256227577 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140402487 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33633620 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514953073679872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115798744 _MDev(–ÝÛiceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6635745845760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451276862841392 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69377424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6325105778062 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6349757634048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚514953097619968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6635972530688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885312303523407e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140428831 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457602893633818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473033289325568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚466683531691520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521588819525632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784Å%ÂvF2·- _wandbruntime‚6614  _runtime‚6614„‚ _'¸€N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103029511 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457682103487276 _runtime‚6614.869754624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6349969122816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6350879107584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140453688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473116029884928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6636931739136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6350879107584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 % _timestamp‚1.7885312313338006e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140453688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103028891 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515043299384320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515043323324416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473116029884928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69389489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6349969122816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33640022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33639404 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466765132774400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140427318 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451355897475296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473115101897216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466765150777344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521680231123456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451355880435936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115818906 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256246222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457683028501848 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ó[]M '¹…N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457683028501848 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103029511 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457682103487276 _runtime‚6614.869754624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6349969122816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6350879107584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140453688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473116029884928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6636931739136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6350879107584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 % _timestamp‚1.7885312313338006e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140453688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103028891 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515043299384320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515043323324416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473116029884928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69389489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6349969122816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33640022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33639404 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466765132774400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140427318 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451355897475296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473115101897216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466765150777344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521680231123456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451355880435936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115818906 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256246222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008Ƨ6:82º- _wandbruntime‚6614  _runtime‚6614£õɽV»Ð 0loss/FBt2/assignment_loss‚0.2035579464281856 /loss/FRt1/detection_loss‚3.4351332017353604 0loss/SRqqt1/detection_loss‚2.254271060228348 /loss/SRqqt4/detection_loss‚2.31725517967168 % _timestamp‚1.7885312313361497e+09 _runtime‚6614.871464452 0loss/FRt3/assignment_loss‚1.1238469996235587 0loss/FBt1/assignment_loss‚0.2520713119626938 1loss/SRqqt3/detection_loss‚0.8535062221368563 .loss/FBt1/detection_loss‚2.523825748761495 /loss/FBt4/detection_loss‚2.3453496439116344 epoch‚4 _step‚455 1loss/SRqqt2/detection_loss‚2.7186540035264835 .loss/FBt2/detection_loss‚3.501526065170765 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚1.8033738215935642 2loss/SRqqt3/assignment_loss‚0.9217378558535801 0loss/FBt3/assignment_loss‚0.1266115690848467 /loss/FBt3/detection_loss‚0.4510887597492938 %loss/total_loss‚44.91094363207152 /loss/FRt2/assignment_loss‚4.182561091838345 0loss/FBt4/assignment_loss‚0.2796046324073619 /loss/FRt1/assignment_loss‚3.101780301048642 2loss/SRqqt2/assignment_loss‚1.6490926491096616 /loss/FRt2/detection_loss‚2.3123729790114016 .loss/FRt4/detection_loss‚2.170028546339144 2loss/SRqqt4/assignment_loss‚1.6222624454428167 /loss/FRt3/detection_loss‚1.5080368641312376 /loss/FRt4/assignment_loss‚3.253394733304563‹ ðË[¼Õ 0loss/FRt3/assignment_loss‚1.1238469996235587 0loss/FBt1/assignment_loss‚0.2520713119626938 1loss/SRqqt3/detection_loss‚0.8535062221368563 .loss/FBt1/detection_loss‚2.523825748761495 /loss/FBt4/detection_loss‚2.3453496439116344 epoch‚4 _step‚455 1loss/SRqqt2/detection_loss‚2.7186540035264835 .loss/FBt2/detection_loss‚3.501526065170765 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚1.8033738215935642 2loss/SRqqt3/assignment_loss‚0.9217378558535801 0loss/FBt3/assignment_loss‚0.1266115690848467 /loss/FBt3/detection_loss‚0.4510887597492938 %loss/total_loss‚44.91094363207152 /loss/FRt2/assignment_loss‚4.182561091838345 0loss/FBt4/assignment_loss‚0.2796046324073619 /loss/FRt1/assignment_loss‚3.101780301048642 2loss/SRqqt2/assignment_loss‚1.6490926491096616 /loss/FRt2/detection_loss‚2.3123729790114016 .loss/FRt4/detection_loss‚2.170028546339144 2loss/SRqqt4/assignment_loss‚1.6222624454428167 /loss/FRt3/detection_loss‚1.5080368641312376 /loss/FRt4/assignment_loss‚3.253394733304563 0loss/FBt2/assignment_loss‚0.2035579464281856 /loss/FRt1/detection_loss‚3.4351332017353604 0loss/SRqqt1/detection_loss‚2.254271060228348 /loss/SRqqt4/detection_loss‚2.31725517967168 % _timestamp‚1.7885312313361497e+09 _runtime‚6614.871464452DZVŽ‚(:… Ÿ¤ëÔ£³á gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.7' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Z¢b¾‚(:× Ÿ¤ëÔÚ¤èá disk.md0.in‚ 199.5859375 disk.md0.out‚ 1166.2890625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚2.125# proc.memory.rssMB‚ 2984.33203125 disk.dm-0.in‚ 9.1640625) disk./.usagePercent‚75.95281613243108 disk.sdb2.in‚ 70.9609375! disk.nvme0n1.out‚ 78458.390625* proc.memory.percent‚18.214917182922363 network.recv‚ 2.220074e+06 disk.sda.out‚1192.7568359375 disk.rbd0.in‚0 disk.sdb.in‚ 70.9609375& proc.memory.availableMB‚ 1207.03125 disk.sdb.out‚1192.7568359375$ memory_percent‚92.63286590576172 cpu‚25.21203041852189 disk.dm-0.out‚103334.7109375! disk.sdb2.out‚1192.7568359375 disk.sda.in‚128.625 disk.md0p1.out‚ 1166.2890625 disk.rbd0.out‚ 1.962890625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sda2.out‚1192.7568359375$ disk./.usageGB‚2716.440601348877" disk.nvme1n1.out‚ 24876.3203125 disk.nvme0n1.in‚1.484375 disk.sda2.in‚128.625 network.sent‚ 1.4075702e+07:³Ø”¿‚(:‰ ®¤ëÔœ³½à gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ }gÀ‚(:Ü ®¤ëÔÏØà) disk./.usagePercent‚75.95278712076635 disk.sda.in‚128.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0 disk.sdb1.out‚0 network.sent‚ 1.417017e+07 network.recv‚ 2.225114e+06 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.sda.out‚1194.2900390625$ memory_percent‚92.63238906860352! disk.sda2.out‚1194.2900390625 disk.md0p1.in‚ 199.5859375* proc.memory.percent‚18.214917182922363 proc.cpu.threads‚17 disk.sdb.out‚1194.2900390625 disk.rbd2.out‚2.125 disk.nvme0n1.out‚ 78696.53125' proc.memory.availableMB‚ 1207.109375 disk.md0p1.out‚ 1167.76171875 disk.sdb1.in‚0# proc.memory.rssMB‚ 2984.33203125 cpu‚25.211599005790653 disk.rbd0.out‚ 1.962890625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1194.2900390625# disk.nvme1n1.out‚24876.50390625% disk./.usageGB‚2716.4395637512207 disk.md0.out‚ 1167.76171875 disk.dm-0.in‚ 9.1640625! disk.dm-0.out‚103573.03515625 disk.sdb.in‚ 70.9609375 disk.nvme0n1.in‚1.484375wð“Á‚(:ˆ ½¤ëÔïË“à gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜$Îk‚(:à ½¤ëÔÞñà" disk.nvme0n1.out‚ 78863.3203125 network.recv‚ 2.229487e+06 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375% disk./.usageGB‚2716.4395637512207) disk./.usagePercent‚75.95278712076635 disk.md0.out‚ 1169.07421875 disk.sdb.out‚1195.6650390625# disk.nvme1n1.out‚24876.70703125$ memory_percent‚92.63396263122559( proc.memory.availableMB‚ 1206.8515625 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sda2.out‚1195.6650390625! disk.dm-0.out‚103740.02734375 disk.sda.out‚1195.6650390625* proc.memory.percent‚18.214917182922363 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 disk.nvme0n1.in‚1.484375 disk.sda.in‚128.625 disk.md0p1.out‚ 1169.07421875# proc.memory.rssMB‚ 2984.33203125 cpu‚25.211196560704188 disk.rbd2.out‚2.125 network.sent‚ 1.4178494e+07 disk.rbd2.in‚0 disk.dm-0.in‚ 9.1640625 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1195.6650390625 disk.md0.in‚ 199.5859375 proc.cpu.threads‚17’m¬²Ã‚(:‡ ̤ëÔ‹‹µà gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0,ñ4Žà gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_u;±`Ä‚(:Õ ̤ëÔƒÄÛá( proc.memory.availableMB‚ 1206.8515625 disk.md0.in‚ 199.5859375 disk.sdb2.out‚ 1197.89453125 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.in‚0 disk.sda2.out‚ 1197.89453125* proc.memory.percent‚18.214917182922363) disk./.usagePercent‚75.95628291970662 disk.sda1.out‚0 disk.rbd2.in‚0% disk./.usageGB‚2716.5645904541016 disk.dm-0.in‚ 9.1640625 disk.sdb.out‚ 1197.89453125# disk.nvme1n1.out‚24993.93359375 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125 disk.sda.in‚128.625 disk.md0p1.out‚ 1171.2265625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚128.625 proc.cpu.threads‚17 disk.sda.out‚ 1197.89453125 network.sent‚ 1.4184678e+07 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375$ memory_percent‚92.63396263122559 disk.md0.out‚ 1171.2265625! disk.dm-0.out‚104144.94140625 disk.rbd0.out‚ 1.962890625 disk.sda1.in‚0 disk.rbd2.out‚2.125 disk.nvme0n1.in‚1.484375 cpu‚25.21082817757188 network.recv‚ 2.233958e+06" disk.nvme0n1.out‚ 79151.0078125³2Å- _wandbruntime‚6670  _runtime‚6670e}YÏÈÆ ' lr-AdamW/pg1‚0.00047518460030960706 ' lr-AdamW/pg2‚0.00047518460030960706 trainer/global_step‚5599 % _timestamp‚1.7885312866826108e+09 _runtime‚ 6670.21823288 _step‚456Ç|ÿÍÇÇ trainer/global_step‚5599 % _timestamp‚1.7885312866826108e+09 _runtime‚ 6670.21823288 _step‚456 ' lr-AdamW/pg1‚0.00047518460030960706 ' lr-AdamW/pg2‚0.00047518460030960706È1úÁi2È- _wandbruntime‚6670  _runtime‚66705·‚èü'ÉöO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477169403067392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6404935087616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258454133 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6405845059072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116806744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461609650616126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33925745 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519464382842368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116806746 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885312866837864e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477170331041792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526159347356160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69980672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694739926016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455227742568592 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚6670.219838037 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141647389 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116806746 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380983033170 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116806744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405845059072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477169403067392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470764467979776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141647389 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519464358902272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694964513792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455227759607952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141673733 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526159098828288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185j>²(ÊûO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141647389 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116806746 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380983033170 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116806744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405845059072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477169403067392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470764467979776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141647389 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519464358902272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694964513792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455227759607952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141673733 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526159098828288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477169403067392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6404935087616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258454133 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6405845059072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116806744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461609650616126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33925745 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519464382842368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116806746 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885312866837864e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477170331041792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526159347356160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69980672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694739926016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455227742568592 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚6670.219838037 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368É•†812Ë- _wandbruntime‚6671  _runtime‚6671M£ñ'ÌÿM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519554584606720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6382100306448 _step‚458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141672220 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406056547840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470846105068544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103924304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519554608546816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6695912122368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258499124 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526250496729088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 $ _timestamp‚1.788531287727066e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103924933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455306760163136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116826904 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141698590 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258525496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477252143613440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0|¸3{FÍ„N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477252143613440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519554584606720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6382100306448 _step‚458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141672220 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406056547840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470846105068544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103924304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519554608546816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6695912122368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258499124 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526250496729088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 $ _timestamp‚1.788531287727066e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103924933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455306760163136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116826904 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141698590 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258525496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944Ê “î2Î- _wandbruntime‚6671  _runtime‚6671ªÃÙVÏÐ /loss/FRt1/assignment_loss‚3.207783842908925 2loss/SRqqt2/assignment_loss‚1.3028049709068403 0loss/FBt1/assignment_loss‚0.1729799496368893 1loss/FBt2/assignment_loss‚0.18689931057289044 0loss/FBt4/assignment_loss‚0.7186773754569408 %loss/total_loss‚41.05715070486664 /loss/FBt1/detection_loss‚2.4740592360496523 1loss/SRqqt3/assignment_loss‚0.927797488295115 2loss/SRqqt4/assignment_loss‚1.4150761083282273 $ _timestamp‚1.788531287728766e+09 1loss/FBt3/assignment_loss‚0.05515643436852392 -loss/FRt4/detection_loss‚2.05676402296457 0loss/FBt3/detection_loss‚0.41423604544252157 trainer/global_step‚5599 _step‚459 2loss/SRqqt1/assignment_loss‚1.5614905294619108 .loss/FRt1/detection_loss‚2.775204010050872 0loss/SRqqt1/detection_loss‚2.255122631003982 1loss/SRqqt4/detection_loss‚2.4099055662237365 .loss/FBt4/detection_loss‚2.291871143711938 /loss/FRt2/assignment_loss‚3.185488083145835 0loss/FRt4/assignment_loss‚3.3237723542584314 .loss/FRt2/detection_loss‚3.094472181164857 -loss/FRt3/detection_loss‚1.24079408040509 0loss/FRt3/assignment_loss‚0.9920054865189087 epoch‚4 1loss/SRqqt2/detection_loss‚1.8730149649911456 1loss/SRqqt3/detection_loss‚0.8489098634652712 /loss/FBt2/detection_loss‚2.2728650255335703 _runtime‚6671.264080683Œ´‰–[ÐÕ /loss/FBt1/detection_loss‚2.4740592360496523 1loss/SRqqt3/assignment_loss‚0.927797488295115 2loss/SRqqt4/assignment_loss‚1.4150761083282273 $ _timestamp‚1.788531287728766e+09 1loss/FBt3/assignment_loss‚0.05515643436852392 -loss/FRt4/detection_loss‚2.05676402296457 0loss/FBt3/detection_loss‚0.41423604544252157 trainer/global_step‚5599 _step‚459 2loss/SRqqt1/assignment_loss‚1.5614905294619108 .loss/FRt1/detection_loss‚2.775204010050872 0loss/SRqqt1/detection_loss‚2.255122631003982 1loss/SRqqt4/detection_loss‚2.4099055662237365 .loss/FBt4/detection_loss‚2.291871143711938 /loss/FRt2/assignment_loss‚3.185488083145835 0loss/FRt4/assignment_loss‚3.3237723542584314 .loss/FRt2/detection_loss‚3.094472181164857 -loss/FRt3/detection_loss‚1.24079408040509 0loss/FRt3/assignment_loss‚0.9920054865189087 epoch‚4 1loss/SRqqt2/detection_loss‚1.8730149649911456 1loss/SRqqt3/detection_loss‚0.8489098634652712 /loss/FBt2/detection_loss‚2.2728650255335703 _runtime‚6671.264080683 /loss/FRt1/assignment_loss‚3.207783842908925 2loss/SRqqt2/assignment_loss‚1.3028049709068403 0loss/FBt1/assignment_loss‚0.1729799496368893 1loss/FBt2/assignment_loss‚0.18689931057289044 0loss/FBt4/assignment_loss‚0.7186773754569408 %loss/total_loss‚41.05715070486664Ë1…ëÌ’Ñ‚(:‡ Û¤ëԟ״á gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.638' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.4095 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‡á.lÒ‚(:á Û¤ëÔö˸â disk.rbd2.out‚2.125# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.in‚0$ memory_percent‚92.63432025909424 disk.sdb.in‚ 70.9609375 disk.sda.out‚1199.3486328125 disk.nvme1n1.in‚ 7.6796875# disk.nvme1n1.out‚24994.13671875 disk.md0p1.out‚ 1172.60546875 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 network.recv‚ 2.236911e+06 disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 proc.cpu.threads‚17! disk.sda2.out‚1199.3486328125 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.962890625 disk.nvme0n1.in‚1.484375% disk./.usageGB‚2716.4395637512207" disk.nvme0n1.out‚ 79425.7265625) proc.memory.availableMB‚ 1206.79296875* proc.memory.percent‚18.214917182922363 cpu‚25.210462372990374 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1172.60546875 disk.dm-0.in‚ 9.1640625 disk.sdb.out‚1199.3486328125 network.sent‚ 1.4200537e+07) disk./.usagePercent‚75.95278712076635! disk.dm-0.out‚104419.86328125! disk.sdb2.out‚1199.3486328125 disk.sdb1.in‚07ëí:”Ó‚(:‰ ê¤ëÔÉÒíÞ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úŠjÔ‚(:ß ê¤ëÔª¿à! disk.sda2.out‚1200.8232421875 disk.dm-0.in‚ 9.1640625 disk.sdb.out‚1200.8232421875) proc.memory.availableMB‚ 1206.79296875 proc.cpu.threads‚17 network.sent‚ 1.4294204e+07 network.recv‚ 2.241728e+06 disk.sda.in‚128.625 disk.rbd0.in‚0" disk.nvme1n1.out‚ 24994.3359375 disk.rbd2.out‚2.125 disk.md0.in‚ 199.5859375) disk./.usagePercent‚75.93797239949677% disk./.usageGB‚2715.9097175598145 disk.sdb1.out‚0 disk.nvme0n1.in‚1.484375# disk.nvme0n1.out‚79680.64453125 disk.sdb.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1174.0234375 disk.sdb1.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.214917182922363! disk.sdb2.out‚1200.8232421875 disk.rbd0.out‚ 1.962890625 disk.sda2.in‚128.625 disk.sda.out‚1200.8232421875 disk.md0.out‚ 1174.0234375 disk.sda1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚104674.98046875$ memory_percent‚92.63432025909424 cpu‚25.210081900854398 disk.nvme1n1.in‚ 7.6796875ÚæYÔ”Õ‚(:‰ ù¤ëÔ®êäà gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òjºõhÖ‚(:Ý ù¤ëÔ­ßúá proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2984.33203125 network.sent‚ 1.4303157e+07 disk.sdb.in‚ 70.9609375 disk.nvme0n1.in‚1.484375) proc.memory.availableMB‚ 1207.03515625* proc.memory.percent‚18.214917182922363 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0 disk.md0.out‚ 1175.7109375! disk.dm-0.out‚104887.16796875 disk.sdb.out‚1202.5771484375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375$ memory_percent‚92.63284206390381 cpu‚25.20960207248615! disk.sdb2.out‚1202.5771484375 disk.md0p1.out‚ 1175.7109375 disk.sdb1.in‚0 disk.rbd2.out‚2.125# disk.nvme0n1.out‚79892.68359375 disk.sda.in‚128.625! disk.nvme1n1.out‚ 24994.484375 disk.sdb1.out‚0 disk.sda2.in‚128.625! disk.sda2.out‚1202.5771484375 disk.sda.out‚1202.5771484375 disk.nvme1n1.in‚ 7.6796875 disk.rbd0.in‚0 network.recv‚ 2.247136e+06) disk./.usagePercent‚75.93796471993845% disk./.usageGB‚2715.9094429016113 disk.dm-0.in‚ 9.1640625 disk.md0.in‚ 199.58593752j7[“ׂ(:ˆ ˆ¥ëÔ•ÓÞ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äd;TeØ‚(:Ú ˆ¥ëÔÓÙñà% disk./.usageGB‚2715.8527641296387 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚ 7.6796875 disk.sda1.in‚0* proc.memory.percent‚18.214917182922363 disk.md0p1.out‚ 1177.34375 network.sent‚ 1.4309301e+07 network.recv‚ 2.251402e+06 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.nvme0n1.in‚1.484375" disk.nvme1n1.out‚ 24994.8984375 cpu‚25.209144243884648 disk.md0.out‚ 1177.34375! disk.sdb.out‚1204.28271484375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2984.33203125 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375) disk./.usagePercent‚75.93637995775218 disk.nvme0n1.out‚ 80050.78125 disk.dm-0.in‚ 9.1640625" disk.sdb2.out‚1204.28271484375! disk.sda.out‚1204.28271484375 disk.rbd0.out‚ 1.962890625$ memory_percent‚92.63434410095215( proc.memory.availableMB‚ 1206.7890625 disk.rbd2.out‚2.125" disk.sda2.out‚1204.28271484375 disk.md0.in‚ 199.5859375 disk.dm-0.out‚105045.6796875 proc.cpu.threads‚17(LØõ2Ù- _wandbruntime‚6726  _runtime‚6726NÈøÉÚà trainer/global_step‚5649 % _timestamp‚1.7885313430137765e+09 _runtime‚6726.549274391 _step‚460 ' lr-AdamW/pg1‚0.00047013231300191486 ' lr-AdamW/pg2‚0.00047013231300191486ïƒÐ€ÎÛÈ trainer/global_step‚5649 % _timestamp‚1.7885313430137765e+09 _runtime‚6726.549274391 _step‚460 ' lr-AdamW/pg1‚0.00047013231300191486 ' lr-AdamW/pg2‚0.00047013231300191486Ì´Õs*2Ü- _wandbruntime‚6726  _runtime‚6726£úü'ÝöO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205713408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚481307372758016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104800946 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530729277855744 trainer/global_step‚5649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6461022512640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481307372758016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34217646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474845440273920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474845440273920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142892291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459178639335152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885313430146213e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6461022512640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753609790976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142892291 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481306444783616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6461932484096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6753404077568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34217024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474845422270976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117814744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚465615482584070 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70583922 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229653504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530729048202240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523975668064768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459178622295792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6461932484096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6753404077568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34217024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474845422270976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117814744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚465615482584070 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70583922 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229653504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530729048202240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocatcÜ5i ed‚523975668064768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459178622295792 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6461022512640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753609790976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142892291 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481306444783616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459257656929696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117834906 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104818848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481390113317376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142943492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6438885536768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104819457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70595987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚6727.701974463 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117834904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142943492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70595989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34223468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754591211008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885313441642768e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530820461040128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6463053957632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142917122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚524065893769216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459257656929696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117834906 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104818848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481390113317376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142943492 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6438885536768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104819457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70595987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚6727.701974463 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117834904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142943492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70595989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34223468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754591211008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885313441642768e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530820461040128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6463053957632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142917122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚524065893769216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152ÎT?w~2â- _wandbruntime‚6727  _runtime‚6727†p/XZãÔ 0loss/FRt3/assignment_loss‚1.2792126655578613 2loss/SRqqt1/assignment_loss‚1.8174597246177269 &loss/total_loss‚40.735256311604374 2loss/SRqqt3/assignment_loss‚0.8505231773095399 /loss/FRt3/detection_loss‚1.3208958377440772 /loss/FBt3/detection_loss‚0.5742176101520294 % _timestamp‚1.7885313441679046e+09 _step‚463 trainer/global_step‚5649 1loss/FBt2/assignment_loss‚0.42071681095876556 1loss/FBt3/assignment_loss‚0.07832991003175122 1loss/FBt1/assignment_loss‚0.24753527764344957 1loss/SRqqt2/detection_loss‚2.1329197377659552 0loss/FBt4/assignment_loss‚0.1027518595709201 0loss/SRqqt3/detection_loss‚0.941440847060033 /loss/FBt1/detection_loss‚1.7911712765693664 .loss/FBt2/detection_loss‚2.779347702860832 .loss/FBt4/detection_loss‚2.432001312573751 _runtime‚ 6727.70338773 .loss/FRt4/assignment_loss‚3.15984362972026 1loss/SRqqt1/detection_loss‚2.3178643304290194 .loss/FRt1/detection_loss‚2.903209395133532 .loss/FRt2/detection_loss‚2.357604573482656 0loss/FRt1/assignment_loss‚2.5532784461975098 0loss/FRt2/assignment_loss‚3.2218711495399477 2loss/SRqqt2/assignment_loss‚1.5918328916264135 2loss/SRqqt4/assignment_loss‚1.1906431418012928 .loss/FRt4/detection_loss‚2.297231056404357 1loss/SRqqt4/detection_loss‚2.3733539468533285 epoch‚4AïZ|_äÙ 0loss/FRt3/assignment_loss‚1.2792126655578613 2loss/SRqqt1/assignment_loss‚1.8174597246177269 &loss/total_loss‚40.735256311604374 2loss/SRqqt3/assignment_loss‚0.8505231773095399 /loss/FRt3/detection_loss‚1.3208958377440772 /loss/FBt3/detection_loss‚0.5742176101520294 % _timestamp‚1.7885313441679046e+09 _step‚463 trainer/global_step‚5649 1loss/FBt2/assignment_loss‚0.42071681095876556 1loss/FBt3/assignment_loss‚0.07832991003175122 1loss/FBt1/assignment_loss‚0.24753527764344957 1loss/SRqqt2/detection_loss‚2.1329197377659552 0loss/FBt4/assignment_loss‚0.1027518595709201 0loss/SRqqt3/detection_loss‚0.941440847060033 /loss/FBt1/detection_loss‚1.7911712765693664 .loss/FBt2/detection_loss‚2.779347702860832 .loss/FBt4/detection_loss‚2.432001312573751 _runtime‚ 6727.70338773 .loss/FRt4/assignment_loss‚3.15984362972026 1loss/SRqqt1/detection_loss‚2.3178643304290194 .loss/FRt1/detection_loss‚2.903209395133532 .loss/FRt2/detection_loss‚2.357604573482656 0loss/FRt1/assignment_loss‚2.5532784461975098 0loss/FRt2/assignment_loss‚3.2218711495399477 2loss/SRqqt2/assignment_loss‚1.5918328916264135 2loss/SRqqt4/assignment_loss‚1.1906431418012928 .loss/FRt4/detection_loss‚2.297231056404357 1loss/SRqqt4/detection_loss‚2.3733539468533285 epoch‚4ÏYÌB+”å‚(:‰ —¥ëÔÜ¿Ðß gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û{o¤bæ‚(:× —¥ëÔ ´Âà disk.rbd0.out‚ 1.962890625 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1178.671875! disk.sda.out‚1205.66552734375$ memory_percent‚92.63584613800049 cpu‚25.20860907144767 proc.cpu.threads‚17 disk.nvme1n1.out‚ 24995.09375 disk.sda1.out‚0 disk.nvme0n1.out‚ 80137.40625 disk.sda2.in‚128.625 disk.dm-0.in‚ 9.16796875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 network.sent‚ 1.4312529e+07% disk./.usageGB‚2706.9306449890137 disk.rbd2.out‚2.125" disk.sdb2.out‚1205.66552734375 disk.sda.in‚128.625 disk.md0p1.out‚ 1178.671875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1.48828125! disk.sdb.out‚1205.66552734375# proc.memory.rssMB‚ 2984.33203125) disk./.usagePercent‚75.68691377238336 disk.sda1.in‚0" disk.sda2.out‚1205.66552734375 network.recv‚ 2.255187e+06 disk.rbd2.in‚0) proc.memory.availableMB‚ 1206.54296875 disk.dm-0.out‚105132.5 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.214917182922363ý¦¿¹’ç‚(:‡ ¦¥ëÔÔ°«à gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ=¢ðè‚(:Ý ¦¥ëÔ‘üà disk.sdb2.in‚ 70.9609375 disk.sda2.in‚128.625" disk.sda2.out‚1207.42529296875 cpu‚25.20818044844233 network.recv‚ 2.259816e+06 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.md0.out‚ 1180.359375$ disk./.usageGB‚2706.995719909668" disk.sdb2.out‚1207.42529296875 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0$ memory_percent‚92.63455867767334* proc.memory.percent‚18.214917182922363 disk.rbd2.out‚2.125 network.sent‚ 1.4419532e+07! disk.dm-0.out‚105389.47265625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.in‚128.625 disk.nvme1n1.out‚ 24995.375# disk.nvme0n1.out‚80394.09765625) proc.memory.availableMB‚ 1206.75390625 proc.cpu.threads‚17! disk.sdb.out‚1207.42529296875 disk.nvme0n1.in‚ 1.48828125( disk./.usagePercent‚75.6887332944008 disk.dm-0.in‚ 9.16796875! disk.sda.out‚1207.42529296875 disk.md0p1.out‚ 1180.359375 disk.rbd0.out‚ 1.962890625 diw¾±fxsk.md0.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0QÈœz”é‚(:‰ µ¥ëÔÓ›´à gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µê¿àlê‚(:á µ¥ëÔ²žá# disk.nvme0n1.out‚80616.41015625* proc.memory.percent‚18.214917182922363! disk.sdb2.out‚1209.3369140625 disk.md0p1.out‚ 1182.21484375 disk.rbd2.in‚0! disk.dm-0.out‚105611.96484375 cpu‚25.207792098252483( disk./.usagePercent‚75.6939902720475$ disk./.usageGB‚2707.183734893799 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚2.125 disk.sda2.in‚128.625 network.sent‚ 1.442577e+07 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1209.3369140625! disk.sda2.out‚1209.3369140625) proc.memory.availableMB‚ 1206.97265625# proc.memory.rssMB‚ 2984.33203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1209.3369140625 disk.sda1.in‚0 disk.md0.in‚ 199.5859375$ memory_percent‚92.63322353363037 proc.cpu.threads‚17 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.md0.out‚ 1182.21484375 disk.dm-0.in‚ 9.16796875 disk.rbd0.out‚ 1.962890625 disk.nvme0n1.in‚ 1.48828125 network.recv‚ 2.264455e+06 disk.nvme1n1.in‚ 7.6796875" disk.nvme1n1.out‚ 24995.5546875 disk.sdb1.out‚0kê@†”ë‚(:‰ Ä¥ëÔËÂÞ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æâ©ÿhì‚(:Ý Ä¥ëÔà›æà disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1183.7734375 disk.sdb1.out‚0 network.recv‚ 2.268785e+06 disk.sda.in‚128.625 disk.rbd0.out‚ 1.962890625 disk.sda1.out‚0* proc.memory.percent‚18.214917182922363) disk./.usagePercent‚75.69397587287567! disk.dm-0.out‚105905.47265625# disk.nvme1n1.out‚24998.97265625 disk.nvme0n1.in‚ 1.48828125 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd2.out‚2.125 disk.sda2.in‚128.625 cpu‚25.20741095827582 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.dm-0.in‚ 9.16796875! disk.sda.out‚1210.95166015625 disk.sda1.in‚0 disk.nvme0n1.out‚80906.5" disk.sda2.out‚1210.95166015625 network.sent‚ 1.4431913e+07$ disk./.usageGB‚2707.183219909668! disk.sdb.out‚1210.95166015625 disk.sdb1.in‚0$ memory_percent‚92.63777732849121( proc.memory.availableMB‚ 1206.2265625# proc.memory.rssMB‚ 2984.33203125 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1183.7734375" disk.sdb2.out‚1210.95166015625 disk.rbd2.in‚0 proc.cpu.threads‚17p\£·2í- _wandbruntime‚6783  _runtime‚6783t Ë.ÆîÀ & lr-AdamW/pg1‚0.0004650583037817084 & lr-AdamW/pg2‚0.0004650583037817084 trainer/global_step‚5699 $ _timestamp‚1.788531400427526e+09 _runtime‚6783.963134895 _step‚464KˆØpËïÅ & lr-AdamW/pg2‚0.0004650583037817084 trainer/global_step‚5699 $ _timestamp‚1.788531400427526e+09 _runtime‚6783.963134895 _step‚464 & lr-AdamW/pg1‚0.0004650583037817084ÐtPQ2ð- _wandbruntime‚6783  _runtime‚6783:`Lü'ñöO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105696781 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528486953287168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34509607 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 trainer/global_step‚5699 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71187172 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485444414474240 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6783.964042639 _step‚465 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485444414474240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6518019909120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6493645518390 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118822744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144137193 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535299305563648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6812121397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485443486499840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478926394565120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118822746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144163537 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105696144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528486929347072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6812352276480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463129519062352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262959937 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118822746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118822744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469622239566378 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478926376562176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6517109937664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885314004284115e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144163537 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478926376562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144IÕ}(òûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262959937 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118822746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118822744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469622239566378 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478926376562176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6517109937664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885314004284115e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144163537 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478926376562176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105696781 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528486953287168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34509607 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 trainer/global_step‚5699 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71187172 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485444414474240 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6783.964042639 _step‚465 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485444414474240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6518019909120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6493645518390 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118822744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144137193 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535299305563648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6812121397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485443486499840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478926394565120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118822746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144163537 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105696144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528486929347072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6812352276480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463129519062352Ñ;‰v2ó- _wandbruntime‚6785  _runtime‚6785Ä !ô€N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528577155051520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚479008013650944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6519141382656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485526227045888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118842906 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885314014707718e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144162024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535390700061696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6813521070080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469702374434200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71199239 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479007995648000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6519141382656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34515257 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚466 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479007995648000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485527155033600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚479008013650944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6518231397888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚469703299448772 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118842904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6493854816664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144188394 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚6785.007530901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144188394 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34514639 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6518231397888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71199237 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6813300689920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105713876 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 LgÖ¾Eù>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 trainer/global_step‚5699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535390455741440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528577178991616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118842906 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463208536656896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144162024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Í0Í 'õ…N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6518231397888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚469703299448772 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118842904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6493854816664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144188394 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚6785.007530901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144188394 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34514639 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6518231397888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71199237 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6813300689920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105713876 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 trainer/global_step‚5699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535390455741440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528577178991616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118842906 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463208536656896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144162024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528577155051520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚479008013650944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6519141382656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485526227045888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118842906 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885314014707718e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144162024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535390700061696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6813521070080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469702374434200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71199239 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479007995648000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6519141382656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34515257 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚466 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479007995648000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485527155033600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚479008013650944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185Ò‡¢"©2ö- _wandbruntime‚6785  _runtime‚6785¶ƒ…=`÷Ú 0loss/FRt4/assignment_loss‚3.6012689815117764 1loss/SRqqt2/detection_loss‚2.2895416347309947 1loss/SRqqt3/detection_loss‚0.8435687162809902 /loss/FBt2/detection_loss‚2.7722070440649986 .loss/FBt1/detection_loss‚2.555904912948608 /loss/FBt3/detection_loss‚0.5888494361215868 /loss/FRt1/assignment_loss‚2.672347795672533 1loss/FBt3/assignment_loss‚0.09463181940680652 1loss/FBt4/assignment_loss‚0.44051765150954286 1loss/SRqqt4/detection_loss‚2.4738233274426955 &loss/total_loss‚41.575232972698245 /loss/FRt3/assignment_loss‚1.351670845411718 1loss/FBt1/assignment_loss‚0.03607376475501951 /loss/FRt2/detection_loss‚2.2989975673890513 1loss/SRqqt1/detection_loss‚2.3023481305156435 epoch‚4 trainer/global_step‚5699 _runtime‚6785.008980319 _step‚467 2loss/SRqqt2/assignment_loss‚1.6933299727970734 2loss/SRqqt4/assignment_loss‚1.3795224045371186 /loss/FRt1/detection_loss‚2.7636249341615815 /loss/FRt3/detection_loss‚1.4020817869750317 .loss/FRt4/detection_loss‚2.094046793663158 0loss/FRt2/assignment_loss‚3.3150040730834007 1loss/FBt2/assignment_loss‚0.07816772347723244 /loss/FBt4/detection_loss‚2.0080941102721472 2loss/SRqqt3/assignment_loss‚0.9180722448974847 2loss/SRqqt1/assignment_loss‚1.6015373010720526 % _timestamp‚1.7885314014734774e+093¥ý­eøß /loss/FRt1/assignment_loss‚2.672347795672533 1loss/FBt3/assignment_loss‚0.09463181940680652 1loss/FBt4/assignment_loss‚0.44051765150954286 1loss/SRqqt4/detection_loss‚2.4738233274426955 &loss/total_loss‚41.575232972698245 /loss/FRt3/assignment_loss‚1.351670845411718 1loss/FBt1/assignment_loss‚0.03607376475501951 /loss/FRt2/detection_loss‚2.2989975673890513 1loss/SRqqt1/detection_loss‚2.3023481305156435 epoch‚4 trainer/global_step‚5699 _runtime‚6785.008980319 _step‚467 2loss/SRqqt2/assignment_loss‚1.6933299727970734 2loss/SRqqt4/assignment_loss‚1.3795224045371186 /loss/FRt1/detection_loss‚2.7636249341615815 /loss/FRt3/detection_loss‚1.4020817869750317 .loss/FRt4/detection_loss‚2.094046793663158 0loss/FRt2/assignment_loss‚3.3150040730834007 1loss/FBt2/assignment_loss‚0.07816772347723244 /loss/FBt4/detection_loss‚2.0080941102721472 2loss/SRqqt3/assignment_loss‚0.9180722448974847 2loss/SRqqt1/assignment_loss‚1.6015373010720526 % _timestamp‚1.7885314014734774e+09 0loss/FRt4/assignment_loss‚3.6012689815117764 1loss/SRqqt2/detection_loss‚2.2895416347309947 1loss/SRqqt3/detection_loss‚0.8435687162809902 /loss/FBt2/detection_loss‚2.7722070440649986 .loss/FBt1/detection_loss‚2.555904912948608 /loss/FBt3/detection_loss‚0.5888494361215868Ó¨¡Íâ”ù‚(:‰ Ó¥ëÔíÿæá gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}hú:nú‚(:ã Ó¥ëÔ¦æä network.recv‚ 2.272471e+06 disk.dm-0.out‚106098.9609375! disk.sda.out‚1212.23779296875# disk.nvme1n1.out‚24999.19140625 disk.md0p1.in‚ 199.5859375 disk.rbd2.out‚2.125 cpu‚25.20696036250244 network.sent‚ 1.4450387e+07" disk.sdb2.out‚1212.23779296875 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚81099.76953125 disk.sda2.in‚128.625! disk.sdb.out‚1212.23779296875 disk.nvme1n1.in‚ 7.6796875 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 1.962890625 disk.rbd2.in‚0$ memory_percent‚92.63412952423096 proc.cpu.threads‚17 disk.dm-0.in‚ 9.16796875 disk.md0p1.out‚ 1185.015625 disk.sdb1.out‚0) proc.memory.availableMB‚ 1206.82421875# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363) disk./.usagePercent‚75.70096576418773" disk.sda2.out‚1212.23779296875 disk.nvme0n1.in‚ 1.48828125 disk.sdb2.in‚ 70.9609375% disk./.usageGB‚2707.4332122802734 disk.md0.out‚ 1185.015625 disk.sda.in‚128.625 disk.sdb1.in‚0Ý(‰õmû‚(:â â¥ëÔ§¥²à% disk./.usageGB‚2707.4332122802734 disk.sda1.out‚0# disk.nvme0n1.out‚81449.46484375 disk.md0.out‚ 1186.80078125 disk.dm-0.in‚ 9.16796875# proc.memory.rssMB‚ 2984.33203125 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.in‚0 disk.nvme0n1.in‚ 1.48828125 disk.sdb.in‚ 70.9609375 network.recv‚ 2.27753e+06 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375! disk.sdb.out‚1214.07958984375! disk.sda.out‚1214.07958984375* proc.memory.percent‚18.214917182922363 disk.rbd2.in‚0# disk.nvme1n1.out‚24999.41015625 network.sent‚ 1.4546945e+07) disk./.usagePercent‚75.70096576418773 disk.rbd0.in‚0 disk.rbd2.out‚2.125" disk.sda2.out‚1214.07958984375 disk.dm-0.out‚ 106448.875" disk.sdb2.out‚1214.07958984375 disk.md0p1.in‚ 199.5859375 disk.sda.in‚128.625 disk.md0p1.out‚ 1186.80078125$ memory_percent‚92.63265132904053 cpu‚25.20654011904263 proc.cpu.threads‚17 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375) proc.memory.availableMB‚ 1207.06640625 disk.rbd0.out‚ 1.962890625÷áô”ü‚(:‰ â¥ëÔ¨¸§à gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.257' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.81425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?l׿ký‚(:à ñ¥ëÔ¨þåß disk.md0.out‚ 1188.43359375 disk.sdb.out‚1215.7802734375 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1215.7802734375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 disk.sda1.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.70096576418773 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.214917182922363 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.rbd2.out‚2.125# proc.memory.rssMB‚ 2984.33203125% disk./.usageGB‚2707.4332122802734 disk.sda1.in‚0 disk.nvme0n1.in‚ 1.48828125 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375) proc.memory.availableMB‚ 1206.67578125 disk.sdb1.out‚0$ memory_percent‚92.63503551483154 proc.cpu.threads‚17! disk.sdb2.out‚1215.7802734375 disk.nvme1n1.in‚ 7.6796875 disk.md0p1.out‚ 1188.43359375 network.recv‚ 2.282426e+06 disk.nvme1n1.out‚ 24999.5625# disk.nvme0n1.out‚81553.62890625! disk.sda2.out‚1215.7802734375 disk.dm-0.in‚ 9.16796875! disk.dm-0.out‚106553.19140625 cpu‚25.20613060598752 network.sent‚ 1.4551015e+07×@„”þ‚(:‰ ñ¥ëÔµ° à gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Çßû”ÿ‚(:‰ €¦ëÔÃûžà gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ°~{d€‚(:Ù €¦ëÔù¼ºà) disk./.usagePercent‚75.70096587084826 disk.sda2.in‚128.625 disk.sda.in‚128.625# memory_percent‚92.6365852355957' proc.memory.availableMB‚ 1206.421875 network.recv‚ 2.286704e+06 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1189.671875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.out‚1217.0751953125 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125! disk.dm-0.out‚106904.47265625# disk.nvme1n1.out‚24999.78515625 disk.sdb1.in‚0 disk.sda1.in‚0 cpu‚25.20574616274419 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.rbd2.out‚2.125! disk.sda2.out‚1217.0751953125% disk./.usageGB‚2707.4332160949707 disk.md0.out‚ 1189.671875 disk.sdb.out‚1217.0751953125 disk.nvme1n1.in‚ 7.6796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 1.48828125 disk.nvme0n1.out‚ 81904.6875* proc.memory.percent‚18.214917182922363 network.sent‚ 1.4559403e+07 disk.dm-0.in‚ 9.16796875! disk.sdb2.out‚1217.0751953125 disk.rbd0.out‚ 1.962890625 disk.sdb1.out‚0 proc.cpu.threads‚17eÎx{2- _wandbruntime‚6841  _runtime‚6841¤4û'Æ‚À & lr-AdamW/pg1‚0.0004599636300649042 & lr-AdamW/pg2‚0.0004599636300649042 trainer/global_step‚5749 $ _timestamp‚1.788531457860672e+09 _runtime‚6841.396193129 _step‚468Ø3•P˃Š& lr-AdamW/pg2‚0.0004599636300649042 trainer/global_step‚5749 $ _timestamp‚1.788531457860672e+09 _runtime‚6841.396193129 _step‚468 & lr-AdamW/pg1‚0.0004599636300649042ÔùWÓ¤2„- _wandbruntime‚6841  _runtime‚6841s‡mü'…öO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119830744 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71790424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489581456190464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532998238509568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145382095 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539869445486080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539869192763904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145382095 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34799261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871206976512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚483007330853376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚483007348856320 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145408439 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71790422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467080398789552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885314578621614e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚483007330853376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119830746 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467080381750192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106589683 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚469 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549522773498 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6574107334144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489580528216064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106590316 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34799892 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473628996548686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473629921563050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6870978194432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489581456190464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489580528216064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548614798494 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6573197362688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6573197362688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574107334144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489580528216064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548614798494 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6573197362688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6573197362688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574107334144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚483007348856320 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145408439 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71790422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467080398789552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885314578621614e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚483007330853376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119830746 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467080381750192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106589683 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚469 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549522773498 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6574107334144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489580528216064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106590316 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34799892 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473628996548686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473629921563050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6870978194432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489581456190464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360Õý<êØ2‡- _wandbruntime‚6842  _runtime‚6842ŠÍ´'ˆ€N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119850906 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106608262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106607652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265257830 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚483088949939200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6575228807680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119850906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚6842.418845302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483088967942144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34805165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539960612526080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34805773 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872172252160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539960846360576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145406926 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489664196749824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489664196749824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6574318822912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145433296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚533088440273920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6575228807680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885314588827274e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489663268762112 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473710056431080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473709131416508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467159416384096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549732071772 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145433296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119850904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145406926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71802487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144êMX '‰…N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533088464214016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚5749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6574318822912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145406926 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489664196749824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489664196749824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6574318822912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145433296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚533088440273920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6575228807680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885314588827274e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489663268762112 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473710056431080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473709131416508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467159416384096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549732071772 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145433296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119850904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145406926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71802487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119850906 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106608262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106607652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265257830 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚483088949939200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6575228807680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119850906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚6842.418845302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483088967942144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34805165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539960612526080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34805773 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872172252160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539960846360576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493717569932288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471031278516752 % _timestamp‚1.7885315151005015e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146626997 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6604492053602 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120838746 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630194759168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544439197773312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537509523731968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493718497906688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146653341 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267465741 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72393674 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493717569932288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107484814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35091142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477636678545358 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493718497906688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267492087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120838744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107485461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35091787 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚477635753530994 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487088303147520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120838746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544439458884096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487088285144576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6630194759168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6929935152128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120838744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72393674 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493717569932288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107484814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35091142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477636678545358 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493718497906688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267492087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120838744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107485461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35091787 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚477635753530994 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487088303147520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120838746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544439458884096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487088285144576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6630194759168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6929935152128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120838744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493717569932288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471031278516752 % _timestamp‚1.7885315151005015e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146626997 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6604492053602 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120838746 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630194759168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544439197773312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537509523731968Ù˜Ñ_Ì2™- _wandbruntime‚6899  _runtime‚6899öbÔ¼š€N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚493800310478336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35096787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885315161907096e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120858904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267510732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267537104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6631316232704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107503175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚493801238466048 _runtime‚6899.727401728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493801238466048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120858906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107502524 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471110296111296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6930860986368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _step‚474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚544530586482688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487169922233344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537599749436416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6605609326880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚477715888398816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146651828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72405739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_s¾EJplit_bytes.small_pool.allocated‚ 6931091852288 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267510732 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚477715888398816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146651828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72405739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6931091852288 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267510732 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885315161907096e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120858904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267510732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267537104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6631316232704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107503175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚493801238466048 _runtime‚6899.727401728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493801238466048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120858906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107502524 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471110296111296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6930860986368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _step‚474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚544530586482688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487169922233344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537599749436416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6605609326880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÚHô2œ- _wandbruntime‚6899  _runtime‚68995°r‹\Ö .loss/FRt2/detection_loss‚2.308187632942023 2loss/SRqqt2/assignment_loss‚1.5599520048973234 0loss/SRqqt3/assignment_loss‚1.10558353865949 /loss/SRqqt1/detection_loss‚2.46568531027207 1loss/SRqqt2/detection_loss‚2.1175251576033505 /loss/FBt1/detection_loss‚2.6372585826449924 0loss/FRt1/assignment_loss‚2.9526008212048076 0loss/FRt4/assignment_loss‚2.9044535694022975 1loss/FBt4/assignment_loss‚0.27177581995056244 _runtime‚6899.728887335 trainer/global_step‚5799 /loss/FRt3/detection_loss‚1.5817895202485264 %loss/total_loss‚43.22083607856244 epoch‚4 /loss/FRt2/assignment_loss‚3.636407821409164 0loss/FRt3/assignment_loss‚1.6586528312964517 0loss/FBt1/assignment_loss‚0.1427005240567875 /loss/FRt1/detection_loss‚3.1341886442640554 1loss/SRqqt4/detection_loss‚2.2846622512890744 2loss/SRqqt1/assignment_loss‚1.7365683999199133 2loss/FBt3/assignment_loss‚0.028389215303571136 /loss/FBt2/detection_loss‚2.1668168207009635 1loss/SRqqt4/assignment_loss‚1.376719467055339 3loss/FBt2/assignment_loss‚0.0070839615576725615 .loss/FRt4/detection_loss‚2.261881460241663 1loss/SRqqt3/detection_loss‚0.8451808471447573 /loss/FBt3/detection_loss‚0.5684642323687898 _step‚475 /loss/FBt4/detection_loss‚3.4683076441287994 $ _timestamp‚1.788531516193495e+09[lË¢ažÛ /loss/FBt4/detection_loss‚3.4683076441287994 $ _timestamp‚1.788531516193495e+09 .loss/FRt2/detection_loss‚2.308187632942023 2loss/SRqqt2/assignment_loss‚1.5599520048973234 0loss/SRqqt3/assignment_loss‚1.10558353865949 /loss/SRqqt1/detection_loss‚2.46568531027207 1loss/SRqqt2/detection_loss‚2.1175251576033505 /loss/FBt1/detection_loss‚2.6372585826449924 0loss/FRt1/assignment_loss‚2.9526008212048076 0loss/FRt4/assignment_loss‚2.9044535694022975 1loss/FBt4/assignment_loss‚0.27177581995056244 _runtime‚6899.728887335 trainer/global_step‚5799 /loss/FRt3/detection_loss‚1.5817895202485264 %loss/total_loss‚43.22083607856244 epoch‚4 /loss/FRt2/assignment_loss‚3.636407821409164 0loss/FRt3/assignment_loss‚1.6586528312964517 0loss/FBt1/assignment_loss‚0.1427005240567875 /loss/FRt1/detection_loss‚3.1341886442640554 1loss/SRqqt4/detection_loss‚2.2846622512890744 2loss/SRqqt1/assignment_loss‚1.7365683999199133 2loss/FBt3/assignment_loss‚0.028389215303571136 /loss/FBt2/detection_loss‚2.1668168207009635 1loss/SRqqt4/assignment_loss‚1.376719467055339 3loss/FBt2/assignment_loss‚0.0070839615576725615 .loss/FRt4/detection_loss‚2.261881460241663 1loss/SRqqt3/detection_loss‚0.8451808471447573 /loss/FBt3/detection_loss‚0.5684642323687898 _step‚475ÛŽ“Ÿ‚(:ˆ ¼¦ëÔ•Ïá gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.50975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¤¹k ‚(:à ¼¦ëÔïô˜ä$ memory_percent‚92.63677597045898' proc.memory.availableMB‚ 1206.390625 cpu‚25.204259989025207 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 network.sent‚ 1.4695482e+07) disk./.usagePercent‚75.68692198524433 disk.nvme0n1.in‚ 1.48828125 disk.sda2.in‚128.625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 7.6796875# disk.nvme1n1.out‚25001.57421875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1195.671875 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0" disk.sda2.out‚1223.30810546875 disk.dm-0.out‚ 107723.484375$ disk./.usageGB‚2706.930938720703 disk.rbd2.out‚2.125 disk.rbd2.in‚0 disk.md0.out‚ 1195.671875 disk.dm-0.in‚ 9.16796875! disk.sdb.out‚1223.30810546875 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125* proc.memory.percent‚18.214917182922363# disk.nvme0n1.out‚82721.91015625" disk.sdb2.out‚1223.30810546875 disk.sda.in‚128.625 disk.sda1.in‚0 network.recv‚ 2.305614e+06! disk.sda.out‚1223.30810546875}—‹a”¡‚(:‰ ˦ëÔ·Ÿà gpu.0.gpu‚13.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.387' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.59675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yd­.s¢‚(:è ˦ëÔæȶà disk.nvme0n1.in‚ 1.48828125 network.sent‚ 1.4789847e+07 network.recv‚ 2.311041e+06) disk./.usagePercent‚75.68700294058821 disk.sda1.in‚0 disk.rbd2.in‚0 cpu‚25.203918894838676" disk.sdb2.out‚1225.13525390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2984.33203125 disk.sda1.out‚0 disk.rbd2.out‚2.125! disk.sdb.out‚1225.13525390625$ memory_percent‚92.63694286346436) proc.memory.availableMB‚ 1206.36328125% disk./.usageGB‚2706.9338340759277# disk.nvme0n1.out‚82934.28515625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 1197.44140625! disk.sda.out‚1225.13525390625 disk.nvme1n1.in‚ 7.6796875" disk.sda2.out‚1225.13525390625 disk.md0.in‚ 199.5859375 disk.rbd0.out‚ 1.962890625* proc.memory.percent‚18.214917182922363 disk.sdb1.in‚0! disk.dm-0.out‚107936.10546875 disk.md0p1.out‚ 1197.44140625 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.sda2.in‚128.625 disk.dm-0.in‚ 9.16796875" disk.nvme1n1.out‚ 25001.8203125òh“£‚(:ˆ Ú¦ëÔÞ”Ÿà gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09e.Ki¤‚(:Þ Ú¦ëÔäØÒà disk.sda1.in‚0 disk.nvme0n1.in‚ 1.48828125 disk.nvme1n1.in‚ 7.6796875 cpu‚25.203549230708134 disk.sdb2.out‚1226.748046875 network.recv‚ 2.315261e+06 disk.sda2.out‚1226.748046875 disk.dm-0.in‚ 9.16796875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0p1.out‚ 1198.9921875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375 network.sent‚ 1.4797693e+07) disk./.usagePercent‚75.69049628633626" disk.nvme0n1.out‚ 83180.3828125 disk.sda.in‚128.625# disk.nvme1n1.out‚25002.12890625# proc.memory.rssMB‚ 2984.33203125 disk.rbd2.out‚2.125 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375! disk.dm-0.out‚108182.51171875 disk.sdb.out‚1226.748046875) proc.memory.availableMB‚ 1206.36328125% disk./.usageGB‚2707.0587730407715 disk.sdb1.out‚0 disk.md0.out‚ 1198.9921875 disk.rbd0.out‚ 1.962890625* proc.memory.percent‚18.214917182922363 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1226.748046875$ memory_percent‚92.63694286346436 disk.md0p1.in‚ 199.5859375LV˜à”¥‚(:‰ é¦ëÔ’¨Àà gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04tŠ¤[¦‚(:Ð é¦ëÔÎæôà% disk./.usageGB‚2707.1837730407715 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 1.52734375 disk.sda2.in‚128.625) disk./.usagePercent‚75.69399133865282 disk.md0.out‚ 1200.3125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 cpu‚25.203180846372426 disk.dm-0.out‚108437.4609375 disk.sdb2.out‚1228.119140625 disk.sda.out‚1228.119140625" disk.nvme1n1.out‚ 25002.3359375 disk.rbd2.in‚0 disk.sda2.out‚1228.119140625# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17 network.sent‚ 1.4801798e+07 disk.md0.in‚ 199.5859375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 disk.sda1.in‚0 disk.dm-0.in‚ 9.20703125 disk.rbd2.out‚2.125 disk.nvme0n1.out‚ 83435.125 network.recv‚ 2.319998e+06 disk.sdb1.in‚0# memory_percent‚92.6384449005127( proc.memory.availableMB‚ 1206.1171875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1228.119140625 disk.sda.in‚128.625 disk.md0p1.out‚ 1200.3125* proc.memory.percent‚18.214917182922363^ ïÝ2§- _wandbruntime‚6955  _runtime‚6955ðsvɨà _step‚476 ' lr-AdamW/pg1‚0.00044971654011615695 ' lr-AdamW/pg2‚0.00044971654011615695 trainer/global_step‚5849 % _timestamp‚1.7885315721088254e+09 _runtime‚6955.644329523ØfaUÎ©È ' lr-AdamW/pg2‚0.00044971654011615695 trainer/global_step‚5849 % _timestamp‚1.7885315721088254e+09 _runtime‚6955.644329523 _step‚476 ' lr-AdamW/pg1‚0.00044971654011615695ÜÜù££2ª- _wandbruntime‚6955  _runtime‚6955öDä6ü'«öO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72996924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474982158243952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6988266554880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147871899 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497854611648512 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121846744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6660369308710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147898243 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481642510513302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6685372212736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491169239435776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚491169257438720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6686282184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542020785014272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885315721098049e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6686282184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491169257438720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6988493239808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497855539622912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269744989 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474982141204592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497854611648512 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497854611648512 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6660369308710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147898243 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481642510513302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6685372212736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491169239435776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚491169257438720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6686282184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542020785014272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885315721098049e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6686282184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491169257438720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6988493239808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497855539622912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269744989 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474982141204592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784ÝW$i2­- _wandbruntime‚6956  _runtime‚6956d¡Ø'®€N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121866906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497937352194560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549100709772800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6989675113984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147923100 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542111034658816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481723570395696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6989471511040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491250876524544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121866904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚5849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491250858521600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚478 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6686493672960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497938280182272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885315732339618e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121866904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6687403657728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203602944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147896730 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108398216 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147896730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542111010718720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497938280182272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108397601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73008989 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73008987 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121866906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6662394557200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269790006 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147923100 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227543040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚549100482229760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615šä[v '¯…N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6686493672960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35389227 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚6956.770329648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147896730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542111010718720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497938280182272 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108397601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73008989 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73008987 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121866906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6662394557200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269790006 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147923100 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227543040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚549100482229760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121866906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497937352194560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549100709772800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6989675113984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147923100 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542111034658816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481723570395696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6989471511040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491250876524544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121866904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚5849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491250858521600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚478 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6686493672960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497938280182272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885315732339618e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121866904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6687403657728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203602944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147896730 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108398216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109275630 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6716246563818 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553579057405440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73600172 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478933020931792 _step‚481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495250211729920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885316293997667e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7047220145152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6741459637760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501992581339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122854746 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546532070236672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485650192509974 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122854744 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149116801 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495250211729920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109275003 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478933037971152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495250193726976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7046987168768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73600174 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚553579314321920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501991653364736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674831 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6742369609216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495250193726976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6717154538822 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6741459637760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149143145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149116801 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122854744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚5899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501991653364736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546532094176768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122854746 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149143145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚7012.939168567 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501992581339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6742369609216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615bO)X(ÀûO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73600174 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚553579314321920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501991653364736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674831 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6742369609216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495250193726976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6717154538822 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6741459637760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149143145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149116801 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122854744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚5899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501991653364736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546532094176768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122854746 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149143145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚7012.939168567 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501992581339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6742369609216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109275630 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6716246563818 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553579057405440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73600172 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478933020931792 _step‚481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495250211729920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885316293997667e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7047220145152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6741459637760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501992581339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122854746 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546532070236672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485650192509974 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122854744 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149116801 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495250211729920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109275003 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478933037971152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495250193726976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7046987168768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640á›+42Á- _wandbruntime‚7013  _runtime‚7013wf›‘c€N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6742581097984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485730327378004 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149141632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7048180002816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149168002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6742581097984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73612237 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553670713972736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7013.925517102 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122874904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149141632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479012038526336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502075321898496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495331830815744 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122874904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚5899 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122874906 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109292885 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615½%8œ 'Ã…N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73612237 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553670713972736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7013.925517102 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122874904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149141632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479012038526336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502075321898496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495331830815744 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122874904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚5899 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122874906 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109292885 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6742581097984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485730327378004 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149141632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7048180002816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149168002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6742581097984â²´ë2Ä- _wandbruntime‚7013  _runtime‚7013½¿šW^ÅØ 0loss/FRt3/assignment_loss‚1.0658062977450236 2loss/SRqqt2/assignment_loss‚1.2955101394782895 /loss/FRt3/detection_loss‚1.4087416837790183 1loss/SRqqt4/detection_loss‚2.2033145499951914 1loss/SRqqt2/detection_loss‚2.1947308402994405 1loss/SRqqt3/detection_loss‚0.7770918184007057 2loss/FBt3/assignment_loss‚0.008646664355113187 .loss/FRt2/detection_loss‚2.541484238975567 epoch‚4 _runtime‚7013.930071512 _step‚483 1loss/FBt1/assignment_loss‚0.22089741400239404 0loss/FBt4/assignment_loss‚0.5323543520049725 /loss/FRt4/detection_loss‚2.0268603996076484 1loss/SRqqt1/detection_loss‚2.8636681834856668 .loss/FBt1/detection_loss‚2.266189530491829 &loss/total_loss‚42.313601595244336 /loss/FBt4/detection_loss‚2.7039740309119225 trainer/global_step‚5899 $ _timestamp‚1.788531630391297e+09 /loss/FRt1/detection_loss‚2.7901692305292403 /loss/FRt1/assignment_loss‚3.665537033762251 1loss/SRqqt4/assignment_loss‚1.732912725910093 1loss/FBt2/assignment_loss‚0.20261495157683385 1loss/SRqqt1/assignment_loss‚1.808409768910635 0loss/FRt2/assignment_loss‚3.0644659951881126 0loss/FRt4/assignment_loss‚3.4127901982753834 2loss/SRqqt3/assignment_loss‚0.9187466061457691 /loss/FBt2/detection_loss‚2.0816704109311104 /loss/FBt3/detection_loss‚0.5270145304821199¿—6cÆÝ 0loss/FRt4/assignment_loss‚3.4127901982753834 2loss/SRqqt3/assignment_loss‚0.9187466061457691 /loss/FBt2/detection_loss‚2.0816704109311104 /loss/FBt3/detection_loss‚0.5270145304821199 0loss/FRt3/assignment_loss‚1.0658062977450236 2loss/SRqqt2/assignment_loss‚1.2955101394782895 /loss/FRt3/detection_loss‚1.4087416837790183 1loss/SRqqt4/detection_loss‚2.2033145499951914 1loss/SRqqt2/detection_loss‚2.1947308402994405 1loss/SRqqt3/detection_loss‚0.7770918184007057 2loss/FBt3/assignment_loss‚0.008646664355113187 .loss/FRt2/detection_loss‚2.541484238975567 epoch‚4 _runtime‚7013.930071512 _step‚483 1loss/FBt1/assignment_loss‚0.22089741400239404 0loss/FBt4/assignment_loss‚0.5323543520049725 /loss/FRt4/detection_loss‚2.0268603996076484 1loss/SRqqt1/detection_loss‚2.8636681834856668 .loss/FBt1/detection_loss‚2.266189530491829 &loss/total_loss‚42.313601595244336 /loss/FBt4/detection_loss‚2.7039740309119225 trainer/global_step‚5899 $ _timestamp‚1.788531630391297e+09 /loss/FRt1/detection_loss‚2.7901692305292403 /loss/FRt1/assignment_loss‚3.665537033762251 1loss/SRqqt4/assignment_loss‚1.732912725910093 1loss/FBt2/assignment_loss‚0.20261495157683385 1loss/SRqqt1/assignment_loss‚1.808409768910635 0loss/FRt2/assignment_loss‚3.0644659951881126ãKEvv”Ç‚(:‰ ´§ëÔÆ¢Éá gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:̓ïmÈ‚(:â ´§ëÔšë†â disk.sdb1.in‚0 disk.rbd2.out‚2.125 disk.sda2.in‚128.625! disk.sda2.out‚1236.1982421875 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 9.20703125* proc.memory.percent‚18.214917182922363 network.recv‚ 2.340745e+06) disk./.usagePercent‚75.68616394884253% disk./.usageGB‚2706.9038276672363 disk.rbd2.in‚0# disk.nvme0n1.out‚84539.60546875# proc.memory.rssMB‚ 2984.33203125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1.52734375 disk.sdb.in‚ 70.9609375$ memory_percent‚92.63768196105957 disk.sdb1.out‚0 disk.nvme1n1.in‚ 7.6796875 cpu‚25.201363885338452 disk.sda.in‚128.625 disk.sda.out‚1236.1982421875# disk.nvme1n1.out‚25004.16015625 disk.sdb.out‚1236.1982421875 disk.sdb2.in‚ 70.9609375( proc.memory.availableMB‚ 1206.2421875! disk.sdb2.out‚1236.1982421875 disk.sda1.in‚0 disk.md0.out‚ 1208.08984375 disk.md0p1.out‚ 1208.08984375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 network.sent‚ 1.4931995e+07 disk.sda1.out‚0 disk.dm-0.out‚ 109543.765625 disk.md0p1.in‚ 199.58593756½—§”É‚(:‰ çëÔ̧²à gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þp2Ò- _wandbruntime‚7070  _runtime‚7070*R••å$ÓöO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123862746 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110170524 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74203422 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35967102 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6797547062784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110171175 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551043379399168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489656024477918 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚5949 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506128695080960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚499331166021120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558149342639104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6798457034240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482883917698352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123862746 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105738652160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885316865793517e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489656949492282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150388047 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123862744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482883900658992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6773031793930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceS/µpÉtatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274224447 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506129623055360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6798457034240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123862744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150388047 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784D8}F(ÔûO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150361703 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506129623055360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274250793 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35967751 _runtime‚7070.115025363 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506128695080960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚499331166021120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558149342639104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6798457034240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482883917698352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123862746 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105738652160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885316865793517e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489656949492282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150388047 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123862744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482883900658992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6773031793930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274224447 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506129623055360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6798457034240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123862744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150388047 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123862746 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110170524 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74203422 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35967102 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6797547062784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110171175 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551043379399168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489656024477918 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚5949 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506212363614720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6799578507776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123882906 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6773241092204 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110188332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35972845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚499412767104000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107146411520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737084360312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150386534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274269438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506211435627008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274269438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123882904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚7071.216795938 _step‚486 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚558240511389184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35973471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499412767104000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110188960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚551133581163520c"Ô '×…N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482962918253536 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6798668523008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6799578507776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774149067416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274295810 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚499412785106944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885316876805007e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123882904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚7071.216795938 _step‚486 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚558240511389184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35973471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499412767104000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110188960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚551133581163520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482962935292896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7106930225664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499412785106944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150412904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506212363614720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6799578507776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123882906 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6773241092204 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110188332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35972845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚499412767104000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107146411520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737084360312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150386534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274269438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506211435627008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274269438 E“Éäà ' lr-AdamW/pg1‚0.00043421759262639406 ' lr-AdamW/pg2‚0.00043421759262639406 trainer/global_step‚5999 % _timestamp‚1.7885317427164557e+09 _runtime‚7126.252020126 _step‚488"£ìÎåÈ _step‚488 ' lr-AdamW/pg1‚0.00043421759262639406 ' lr-AdamW/pg2‚0.00043421759262639406 trainer/global_step‚5999 % _timestamp‚1.7885317427164557e+09 _runtime‚7126.252020126蜯BÙ2æ- _wandbruntime‚7126  _runtime‚7126–‹°ü'çöO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6853634487808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚486834780386192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124870746 _runtime‚7126.252801289 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510266664771584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7164836189696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151606605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5999 _step‚489 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151632949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6854544459264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6853634487808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124870744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124870744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276503695 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74806674 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151632949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36258958 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562719500811264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885317427171612e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493663706474590 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6828001074034 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555554664621568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510265736797184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510265736797184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74806672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7164613699072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503412120312320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562719254380544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555554640681472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828909049038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151606605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510265736797184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510265736797184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74806672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7164613699072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503412120312320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562719254380544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555554640681472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828909049038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151606605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510266664771584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7164836189696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151606605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5999 _step‚489 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151632949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6854544459264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6853634487808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124870744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124870744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276503695 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74806674 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151632949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36258958 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562719500811264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885317427171612e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493663706474590 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6828001074034 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555554664621568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0驯p2é- _wandbruntime‚7127  _runtime‚7127¹mX«âê€N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151631436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124890906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151631436 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6855665932800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885317437082782e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111084038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74818737 _runtime‚7127.244586371 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151657806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124890904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111083397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493742916328048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6855665932800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151657806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74818739 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7165786796544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124890906 _step‚490 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486913815020096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510349405330944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562810899599872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510349405330944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276522340 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503493739398144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503493739398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36264660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7166009273856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚510348477343232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640TD¬ 'ë…N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510349405330944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276522340 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503493739398144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503493739398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36264660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7166009273856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚510348477343232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151631436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124890906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151631436 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6855665932800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885317437082782e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111084038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74818737 _runtime‚7127.244586371 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151657806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124890904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111083397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493742916328048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6855665932800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151657806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74818739 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7165786796544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124890906 _step‚490 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486913815020096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510349405330944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562810899599872ê5‘¯2ì- _wandbruntime‚7127  _runtime‚7127Ï%~[íÕ 0loss/FRt3/assignment_loss‚1.1793068816264471 0loss/FRt4/assignment_loss‚3.4738462262683445 2loss/SRqqt4/assignment_loss‚1.8990065162220309 1loss/FBt1/assignment_loss‚0.14618641831475543 .loss/FRt1/detection_loss‚3.247534457445145 /loss/FRt2/detection_loss‚2.6119297805590853 .loss/FRt4/detection_loss‚2.373317809899648 0loss/FRt1/assignment_loss‚2.9570540094375612 1loss/SRqqt4/detection_loss‚2.2990722790841134 _step‚491 /loss/FBt2/detection_loss‚2.1025773525238036 2loss/SRqqt2/assignment_loss‚1.1847483344376086 0loss/FBt2/assignment_loss‚0.1572046257885443 2loss/FBt3/assignment_loss‚0.042993317420520545 0loss/SRqqt1/detection_loss‚2.260483150128965 0loss/SRqqt2/detection_loss‚2.004832845926285 1loss/SRqqt3/detection_loss‚0.8301832849780718 trainer/global_step‚5999 /loss/FBt3/detection_loss‚0.9683096105120433 /loss/FRt3/detection_loss‚1.8110769297927618 1loss/SRqqt3/assignment_loss‚0.801487200849113 .loss/FBt4/detection_loss‚3.218554049730301 _runtime‚7127.245629395 % _timestamp‚1.7885317437103157e+09 /loss/FRt2/assignment_loss‚3.068030971905281 2loss/SRqqt1/assignment_loss‚1.3256563376505963 0loss/FBt4/assignment_loss‚0.3574220900173941 .loss/FBt1/detection_loss‚2.046144283734835 &loss/total_loss‚42.366958764253255 epoch‚4Ð[4`îÚ /loss/FBt2/detection_loss‚2.1025773525238036 2loss/SRqqt2/assignment_loss‚1.1847483344376086 0loss/FBt2/assignment_loss‚0.1572046257885443 2loss/FBt3/assignment_loss‚0.042993317420520545 0loss/SRqqt1/detection_loss‚2.260483150128965 0loss/SRqqt2/detection_loss‚2.004832845926285 1loss/SRqqt3/detection_loss‚0.8301832849780718 trainer/global_step‚5999 /loss/FBt3/detection_loss‚0.9683096105120433 /loss/FRt3/detection_loss‚1.8110769297927618 1loss/SRqqt3/assignment_loss‚0.801487200849113 .loss/FBt4/detection_loss‚3.218554049730301 _runtime‚7127.245629395 % _timestamp‚1.7885317437103157e+09 /loss/FRt2/assignment_loss‚3.068030971905281 2loss/SRqqt1/assignment_loss‚1.3256563376505963 0loss/FBt4/assignment_loss‚0.3574220900173941 .loss/FBt1/detection_loss‚2.046144283734835 &loss/total_loss‚42.366958764253255 epoch‚4 0loss/FRt3/assignment_loss‚1.1793068816264471 0loss/FRt4/assignment_loss‚3.4738462262683445 2loss/SRqqt4/assignment_loss‚1.8990065162220309 1loss/FBt1/assignment_loss‚0.14618641831475543 .loss/FRt1/detection_loss‚3.247534457445145 /loss/FRt2/detection_loss‚2.6119297805590853 .loss/FRt4/detection_loss‚2.373317809899648 0loss/FRt1/assignment_loss‚2.9570540094375612 1loss/SRqqt4/detection_loss‚2.2990722790841134 _step‚491ëÚJ •’ï‚(:‡ ¬¨ëÔžéƒà gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒxñTð‚(:É ¬¨ëÔ»ŒÄà disk.dm-0.in‚ 9.20703125! disk.dm-0.out‚111422.80078125 disk.sda.in‚128.625 disk.nvme1n1.in‚ 7.6796875 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 disk.sdb.out‚ 1248.68359375 disk.rbd0.out‚ 1.962890625# disk.nvme0n1.out‚86416.66015625 disk.sda2.in‚128.625 disk.md0.out‚ 1220.109375 cpu‚25.198316278511214 network.recv‚ 2.376231e+06 disk.sda.out‚ 1248.68359375! disk.nvme1n1.out‚ 25006.140625 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2984.33203125 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0) disk./.usagePercent‚75.70097760350679 disk.sdb2.out‚ 1248.68359375 disk.rbd0.in‚0 disk.sda2.out‚ 1248.68359375! memory_percent‚92.63916015625* proc.memory.percent‚18.214917182922363 disk.md0p1.out‚ 1220.109375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.5182406e+07 disk.sdb1.out‚0# disk./.usageGB‚2707.43363571167 disk.sdb.in‚ 70.9609375 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.nvme0n1.in‚ 1.52734375 proc.memory.availableMB‚1206[ÿ“ñ‚(:ˆ »¨ëÔ´¸öÞ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ìêSdò‚(:Ù »¨ëÔûÇ°à disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1.52734375# proc.memory.rssMB‚ 2984.33203125) disk./.usagePercent‚75.70097760350679 disk.sda2.in‚128.625 disk.sda.out‚1250.2431640625 disk.md0p1.out‚ 1221.60546875# disk./.usageGB‚2707.43363571167 disk.md0.out‚ 1221.60546875 disk.dm-0.in‚ 9.20703125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.962890625 network.sent‚ 1.5289309e+07! disk.sdb2.out‚1250.2431640625 disk.sda.in‚128.625 disk.nvme1n1.in‚ 7.6796875 disk.md0.in‚ 199.5859375 disk.sdb.out‚1250.2431640625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.rbd2.out‚2.125 cpu‚25.19797270183141" disk.nvme0n1.out‚ 86757.4609375 disk.sdb2.in‚ 70.9609375$ proc.memory.availableMB‚1205.875* proc.memory.percent‚18.214917182922363 network.recv‚ 2.381143e+06! disk.nvme1n1.out‚ 25006.328125 disk.sdb1.in‚0$ memory_percent‚92.63992309570312 proc.cpu.threads‚17 disk.sda1.out‚0! disk.sda2.out‚1250.2431640625 disk.dm-0.out‚111763.7890625 disk.sdb.in‚ 70.9609375˜è{f’ó‚(:‡ ʨëÔæÞ¿à gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Tpô‚(:å ʨëÔ‹ï½á disk.sdb.in‚ 70.9609375! disk.sdb.out‚1252.08056640625 disk.rbd0.in‚0 disk.sda.in‚128.625 disk.nvme1n1.in‚ 7.6796875# disk.nvme1n1.out‚25006.83203125" disk.sda2.out‚1252.08056640625 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1252.08056640625 cpu‚25.197656091772846 proc.cpu.threads‚17 disk.md0p1.out‚ 1223.38671875) disk./.usagePercent‚75.68615434939464 disk.md0p1.in‚ 199.5859375 disk.rbd2.in‚0 network.recv‚ 2.3849e+06 disk.rbd0.out‚ 1.962890625 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.sdb1.out‚0! disk.sda.out‚1252.08056640625 disk.sda1.in‚0* proc.memory.percent‚18.214917182922363 disk.sdb1.in‚0 disk.nvme0n1.in‚ 1.52734375! disk.dm-0.out‚111855.41015625 network.sent‚ 1.5297143e+07% disk./.usageGB‚2706.9034843444824 disk.sda1.out‚0 disk.md0.out‚ 1223.38671875 disk.dm-0.in‚ 9.20703125$ memory_percent‚92.63842105865479# proc.memory.rssMB‚ 2984.33203125 disk.rbd2.out‚2.125! disk.nvme0n1.out‚ 86848.578125) proc.memory.availableMB‚ 1206.12109375d'¤ç”õ‚(:‰ Ù¨ëÔºòãÞ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ë¬Þkö‚(:à Ù¨ëÔ„…¸à disk.rbd0.out‚ 1.962890625) disk./.usagePercent‚75.68613355059088 disk.md0p1.out‚ 1225.0390625 disk.nvme0n1.in‚ 1.52734375% disk./.usageGB‚2706.9027404785156 disk.dm-0.out‚ 112117.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0# disk.nvme0n1.out‚87110.66015625# proc.memory.rssMB‚ 2984.33203125" disk.sda2.out‚1253.80126953125# disk.nvme1n1.out‚25006.96484375 disk.sda1.in‚0 network.recv‚ 2.396724e+06" disk.sdb2.out‚1253.80126953125 disk.sdb1.in‚0 disk.rbd2.in‚0* proc.memory.percent‚18.214917182922363 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 7.6796875 disk.sda1.out‚0' proc.memory.availableMB‚ 1205.609375 disk.sda2.in‚128.625 disk.md0.out‚ 1225.0390625 disk.sdb1.out‚0 disk.rbd2.out‚2.125 cpu‚25.201587266637514 proc.cpu.threads‚17 disk.dm-0.in‚ 9.20703125! disk.sdb.out‚1253.80126953125 disk.sda.in‚128.625! disk.sda.out‚1253.80126953125$ memory_percent‚92.64154434204102 network.sent‚ 1.5307051e+07 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375V†’÷‚(:‡ è¨ëÔ¿ëôà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Ãª6nø‚(:ã è¨ëÔ‹¦åá$ disk./.usageGB‚2706.888416290283 disk.sda.out‚1255.0712890625 disk.rbd0.in‚0" disk.nvme0n1.out‚ 87191.5859375 disk.nvme1n1.in‚ 7.6796875 disk.sdb1.out‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1205.89453125 network.recv‚ 2.400917e+06 disk.sda1.in‚0 disk.rbd2.out‚2.125# proc.memory.rssMB‚ 2984.33203125 cpu‚25.210295062025914 network.sent‚ 1.5313131e+07) disk./.usagePercent‚75.68573304029277 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚25007.26171875! disk.sda2.out‚1255.0712890625 proc.cpu.threads‚17 disk.rbd0.out‚ 1.962890625 disk.sdb1.in‚0 disk.sda1.out‚0$ memory_percent‚92.63980388641357 disk.sdb.out‚1255.0712890625 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.in‚ 1.52734375 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 9.20703125* proc.memory.percent‚18.214917182922363! disk.sdb2.out‚1255.0712890625 disk.sda.in‚128.625 disk.md0p1.out‚ 1226.23828125 disk.sda2.in‚128.625! disk.dm-0.out‚112198.84765625 disk.sdb2.in‚ 70.9609375 disk.md0.out‚ 1226.2382812582&“ù‚(:ˆ ÷¨ëÔõò­à gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Wuú‚(:ê ÷¨ëÔó¹âà disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1.52734375 disk.dm-0.in‚ 9.20703125$ memory_percent‚92.63896942138672& proc.memory.availableMB‚ 1206.03125 disk.sda.in‚128.625 disk.sda1.in‚0 disk.rbd2.out‚ 2.19921875" disk.sda2.out‚1257.09423828125 proc.cpu.threads‚17 network.sent‚ 1.5316251e+07 network.recv‚ 2.404287e+06! disk.sda.out‚1257.09423828125 disk.nvme1n1.in‚ 7.6796875# proc.memory.rssMB‚ 2984.33203125 cpu‚25.218327870838735) disk./.usagePercent‚75.68697904862901% disk./.usageGB‚2706.9329795837402 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 1.978515625 disk.dm-0.out‚112414.9140625 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.214917182922363" disk.sdb2.out‚1257.09423828125# disk.nvme0n1.out‚87321.23046875 disk.md0p1.out‚ 1228.18359375 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375! disk.sdb.out‚1257.09423828125# disk.nvme1n1.out‚25093.68359375 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1228.18359375FöS“û‚(:ˆ †©ëÔ³¿à gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é €qü‚(:æ †©ëÔÀöãà disk.rbd0.out‚ 2.041015625 disk.sda1.in‚0 disk.sda2.out‚1258.455078125 disk.dm-0.in‚ 16.9140625! disk.dm-0.out‚112608.03515625 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1229.48828125 disk.sda.in‚128.625 disk.sda.out‚1258.455078125) proc.memory.availableMB‚ 1206.02734375 cpu‚25.227112699243705 proc.cpu.threads‚17 network.recv‚ 2.40817e+06) disk./.usagePercent‚75.69047911399058 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.19921875 disk.nvme1n1.in‚ 15.38671875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb.out‚1258.455078125 disk.sdb2.out‚1258.455078125 network.sent‚ 1.5323311e+07% disk./.usageGB‚2707.0581588745117 disk.md0p1.in‚ 199.5859375 disk.sda2.in‚128.625" disk.nvme1n1.out‚ 25129.2734375$ memory_percent‚92.63899326324463* proc.memory.percent‚18.214917182922363 disk.nvme0n1.in‚ 1.52734375# disk.nvme0n1.out‚87478.76171875 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1229.48828125 disk.sdb.in‚ 70.9609375# proc.memory.rssMB‚ 2984.33203125MÈ|2ý- _wandbruntime‚7236  _runtime‚7236šØÿ5 þÝT [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112616637 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 126556428 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚516754840969216 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚509825364700672 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚516755542011392 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6930159307776 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚492940468889232 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112616868 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚562462133221888 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _step‚492 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚229 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚492940485928592 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6930159307776 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7242929552896 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚36786437 KEDeË-“.viceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚509825382703616 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 279765170 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚75830202 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚499844039570954 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚7236.503707388 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0¾x íh*ÿâT JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112616637 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 126556428 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚516754840969216 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚509825364700672 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚516755542011392 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6930159307776 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚492940468889232 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112616868 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚562462133221888 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _step‚492 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚229 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚492940485928592 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6930159307776 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7242929552896 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚36786437 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚509825382703616 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 279765170 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚75830202 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚499844039570954 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚7236.503707388 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570ì2Â<;2€- _wandbruntime‚7237  _runtime‚7237q@Knl)æR `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 178890240 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 126571906 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 153196843 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7243240513536 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 126571906 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 279768747 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚75839762 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚569761844119552 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚493 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6930649158656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 153217554 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚516809591727104 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112631669 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚36791905 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚499894953887022 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7243419403776 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚6044 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚516810292769280 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6929966119424 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚562518400775680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 126571904 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚509879643610624 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚75839764 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚509879625607680 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚36791675 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 202830336 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚562518424715776 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _runtime‚7237.685628059 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112631437 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚230 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚509879643610624 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚569761641289216 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚492990894994416 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6929966119424 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 153196843 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚499895651993830 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚516809591727104 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 126571904 N·§©‚ëR \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚516810292769280 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788531854149272e+09 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 126571904 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚509879643610624 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚75839764 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚509879625607680 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚36791675 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 202830336 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚562518424715776 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _runtime‚7237.685628059 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112631437 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚230 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚509879643610624 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚569761641289216 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚492990894994416 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6929966119424 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 153196843 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚499895651993830 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bï:ò=Èytes.all.current‚ 701042176 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚516809591727104 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 126571904 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 178890240 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 126571906 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 153196843 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7243240513536 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 126571906 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 279768747 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚75839762 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚569761844119552 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚493 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6930649158656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 153217554 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚516809591727104 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112631669 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚36791905 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚499894953887022 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7243419403776 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚6044 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚516810292769280 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6929966119424 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚562518400775680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3946290612220764 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_6‚1 _runtime‚7253.270772278 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638716697693 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4356060326099396 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5037647485733032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940647482872009 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925633668899536 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0911330133676529 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16877014935016632 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 jet/accuracy_1_of_7‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47082245349884033 particle/accuracy_4_of_5‚1 ,particle/sensitivity‚0.35668715834617615 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19993355870246887 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4701991081237793 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24576438963413239 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.014426523819565773 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14073756337165833 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5445050001144409 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01111111231148243 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4436158835887909 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5001187920570374 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1662830412387848 particle/accuracy_3_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4009915590286255 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44387325644493103 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914208650588989 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2549603283405304 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 trainer/global_step‚6044 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0167730413377285 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3511416018009186 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5682330131530762 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 epoch‚4 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4217372536659241 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5399999618530273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08217234909534454 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4889552593231201 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9905086159706116 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41124969720840454 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923250675201416 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5217166543006897 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1540984809398651 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_3_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41807714104652405 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4038672149181366 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19341564178466797 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.53522789478302 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5284130573272705 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991021990776062 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8556031584739685 particle/accuracy_5_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5079397559165955 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2999320924282074 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4889936149120331 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48422425985336304 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32492661476135254 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2181723564863205 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14937181770801544 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5118726491928101 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5173611044883728 jet/accuracy_4_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47550755739212036 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0034936137963086367 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.545634925365448 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5051451325416565 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24559859931468964 particle/accuracy_6_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5036084055900574 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.55729079246521 +jet/accuracy_2_of_2‚0.11280441284179688 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5491584539413452 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5717466473579407 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29353779554367065 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17503829300403595 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5198674201965332 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12081465125083923 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16842308640480042 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 *jet/accuracy_1_of_5‚0.9766832590103149 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898824095726013 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.42007800936698914 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4686186909675598 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023462869226932526 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877253770828247 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24614958465099335 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9873390197753906 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16734011471271515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5662615895271301 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_5‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.36933988332748413 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09002486616373062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_4‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2494014948606491 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.29320988059043884 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4796831011772156 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5352486371994019 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18400901556015015 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5307266116142273 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9848693609237671 particle/accuracy_3_of_4‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚41.48606302341843 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.533991277217865 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.39604488015174866 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.4375 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641730010509491 +particle/specificity‚0.9479448795318604 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5517857670783997 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18333329260349274 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.557440459728241 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 # _timestamp‚1.78853186973266e+09 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1809879094362259 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2937544882297516 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7054968476295471 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008978215046226978 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44555437564849854 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5625587105751038 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25567248463630676 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15964913368225098 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 *jet/accuracy_1_of_1‚0.4909522235393524 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04738863557577133 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2063131332397461 'particle/f_score‚0.4353599548339844 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1824648380279541 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 particyÿRQz3le/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895984530448914 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43535372614860535 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 +jet/accuracy_2_of_3‚0.25799211859703064 jet/accuracy_5_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988794922828674 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009567901492118835 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19710195064544678 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4749158024787903 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.4287036955356598 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3162016272544861 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4094770848751068 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979081153869629 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2466525137424469 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19710195064544678 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49658429622650146 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8755407929420471 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 6"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5078164935112 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 particle/accuracy_1_of_1‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5424283742904663 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020906811580061913 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,jet/accuracy_3_of_3‚0.026894258335232735 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2650969922542572 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22701317071914673 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969731569290161 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143189549446106 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5021227598190308 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3804384171962738 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1114453449845314 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0022564581595361233 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.4791666865348816 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27586203813552856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18913067877292633 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19979840517044067 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.228598490357399 jet/accuracy_3_of_6‚0 particle/accuracy_2_of_7‚1 _step‚494 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5526069402694702 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5181307196617126 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18444447219371796 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1400628685951233 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39717262983322144 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44677305221557617 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5197359323501587 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03916254639625549 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7663134932518005 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9940524697303772 particle/accuracy_5_of_6‚1 particle/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5201793909072876 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 particle/accuracy_3_of_3‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5021227598190308 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5474595427513123 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.44013407826423645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18996423482894897 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5247282385826111 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41328972578048706 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15070682764053345 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.609868586063385 ,jet/accuracy_4_of_4‚0.011393589898943901 *jet/accuracy_2_of_5‚0.5068292617797852 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19084319472312927 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1737416535615921 +jet/accuracy_3_of_4‚0.07943075895309448 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2888888716697693 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5001412034034729 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48165860772132874 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9872311949729919 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.925446629524231 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07261904329061508 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47855091094970703 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02872702293097973 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44655171036720276 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5135351419448853 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16577580571174622 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10290392488241196 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2200738936662674 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17492511868476868 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5047212839126587 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02777777798473835 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29170629382133484 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03285374492406845 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2648738622665405 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06892964988946915 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4961239993572235 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16309525072574615 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_6‚0.5512820482254028 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2171737551689148 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.1303507387638092 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0ÍDëñxL‡Τ 6"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5078164935112 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 particle/accuracy_1_of_1‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5424283742904663 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020906811580061913 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,jet/accuracy_3_of_3‚0.026894258335232735 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2650969922542572 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22701317071914673 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969731569290161 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143189549446106 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5021227598190308 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3804384171962738 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1114453449845314 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0022564581595361233 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.4791666865348816 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27586203813552856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18913067877292633 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19979840517044067 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.228598490357399 jet/accuracy_3_of_6‚0 particle/accuracy_2_of_7‚1 _step‚494 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5526069402694702 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5181307196617126 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18444447219371796 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1400628685951233 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39717262983322144 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44677305221557617 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5197359323501587 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03916254639625549 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7663134932518005 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9940524697303772 particle/accuracy_5_of_6‚1 particle/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5201793909072876 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 particle/accuracy_3_of_3‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5021227598190308 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5474595427513123 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.44013407826423645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18996423482894897 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5247282385826111 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41328972578048706 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15070682764053345 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.609868586063385 ,jet/accuracy_4_of_4‚0.011393589898943901 *jet/accuracy_2_of_5‚0.5068292617797852 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19084319472312927 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1737416535615921 +jet/accuracy_3_of_4‚0.07943075895309448 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2888888716697693 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5001412034034729 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48165860772132874 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9872311949729919 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.925446629524231 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07261904329061508 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47855091094970703 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02872702293097973 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44655171036720276 jet/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5135351419448853 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16577580571174622 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10290392488241196 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2200738936662674 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17492511868476868 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5047212839126587 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02777777798473835 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29170629382133484 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03285374492406845 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2648738622665405 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06892964988946915 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4961239993572235 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16309525072574615 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_6‚0.5512820482254028 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2171737551689148 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.1303507387638092 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028413960710167885 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26144492626190186 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17687667906284332 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5047332644462585 jet/accuracy_7_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5630969405174255 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.003976034931838512 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 *jet/accuracy_2_of_4‚0.4076518714427948 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1453900784254074 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3946290612220764 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_6‚1 _runtime‚7253.270772278 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638716697693 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4356060326099396 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5037647485733032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940647482872009 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925633668899536 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0911330133676529 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16877014935016632 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 jet/accuracy_1_of_7‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47082245349884033 particle/accuracy_4_of_5‚1 ,particle/sensitivity‚0.35668715834617615 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19993355870246887 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4701991081237793 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24576438963413239 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.014426523819565773 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14073756337165833 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5445050001144409 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01111111231148243 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4436158835887909 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5001187920570374 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1662830412387848 particle/accuracy_3_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4009915590286255 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44387325644493103 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914208650588989 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2549603283405304 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 trainer/global_step‚6044 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0167730413377285 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3511416018009186 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5682330131530762 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 epoch‚4 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4217372536659241 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5399999618530273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08217234909534454 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4889552593231201 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9905086159706116 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41124969720840454 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923250675201416 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5217166543006897 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1540984809398651 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_3_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41807714104652405 <&Purÿ+=îÝEity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4038672149181366 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19341564178466797 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.53522789478302 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5284130573272705 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991021990776062 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8556031584739685 particle/accuracy_5_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5079397559165955 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2999320924282074 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4889936149120331 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48422425985336304 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32492661476135254 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2181723564863205 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14937181770801544 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5118726491928101 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5173611044883728 jet/accuracy_4_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47550755739212036 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0034936137963086367 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.545634925365448 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5051451325416565 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24559859931468964 particle/accuracy_6_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5036084055900574 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.55729079246521 +jet/accuracy_2_of_2‚0.11280441284179688 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5491584539413452 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5717466473579407 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29353779554367065 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17503829300403595 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5198674201965332 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12081465125083923 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16842308640480042 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 *jet/accuracy_1_of_5‚0.9766832590103149 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898824095726013 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.42007800936698914 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4686186909675598 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023462869226932526 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877253770828247 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24614958465099335 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9873390197753906 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16734011471271515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5662615895271301 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_5‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.36933988332748413 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09002486616373062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_4‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2494014948606491 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.29320988059043884 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4796831011772156 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5352486371994019 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18400901556015015 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5307266116142273 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9848693609237671 particle/accuracy_3_of_4‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚41.48606302341843 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.533991277217865 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.39604488015174866 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.4375 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641730010509491 +particle/specificity‚0.9479448795318604 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5517857670783997 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18333329260349274 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.557440459728241 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 # _timestamp‚1.78853186973266e+09 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1809879094362259 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2937544882297516 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7054968476295471 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008978215046226978 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44555437564849854 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5625587105751038 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25567248463630676 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15964913368225098 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 *jet/accuracy_1_of_1‚0.4909522235393524 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04738863557577133 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2063131332397461 'particle/f_score‚0.4353599548339844 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1824648380279541 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 particle/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895984530448914 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43535372614860535 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 +jet/accuracy_2_of_3‚0.25799211859703064 jet/accuracy_5_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988794922828674 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009567901492118835 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19710195064544678 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4749158024787903 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.4287036955356598 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3162016272544861 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4094770848751068 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979081153869629 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2466525137424469 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19710195064544678 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49658429622650146 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8755407929420471 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5îHI°Ê’ˆ‚(:‡ ¤©ëÔãä˜à gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ƒÓów‰‚(:ì ¤©ëÔ¹º±à) proc.memory.percent‚18.22059154510498 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1973.78125 cpu‚25.238330723012034) disk./.usagePercent‚75.70097333708551! disk.sda2.out‚1261.9638671875 disk.sda.in‚128.625 disk.sda.out‚1261.9638671875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.md0.out‚ 1232.86328125! disk.dm-0.out‚113807.33984375" disk.nvme1n1.out‚ 25358.2265625 disk.md0p1.out‚ 1232.86328125 disk.rbd0.in‚0 disk.sdb1.out‚0% disk./.usageGB‚2707.4334831237793 disk.dm-0.in‚ 16.9140625 disk.sdb.out‚1261.9638671875# proc.memory.rssMB‚ 2985.26171875 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 15.38671875 network.sent‚ 1.5434979e+07 network.recv‚ 2.418625e+06 disk.rbd2.out‚ 2.30126953125 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 2.12841796875$ memory_percent‚87.95299530029297 disk.nvme0n1.in‚ 1.52734375# disk.nvme0n1.out‚88449.11328125! disk.sdb2.out‚1261.9638671875 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0[˜î™2Š- _wandbruntime‚7264  _runtime‚7264†bEâÄ‹¾ _runtime‚ 7264.52720068 _step‚495 % lr-AdamW/pg1‚0.000429021363290781 % lr-AdamW/pg2‚0.000429021363290781 trainer/global_step‚6049 % _timestamp‚1.7885318809914956e+09ò3‚ûɌà trainer/global_step‚6049 % _timestamp‚1.7885318809914956e+09 _runtime‚ 7264.52720068 _step‚495 % lr-AdamW/pg1‚0.000429021363290781 % lr-AdamW/pg2‚0.000429021363290781ï Y„ƒ2- _wandbruntime‚7264  _runtime‚7264²üydü'ŽöO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 % _timestamp‚1.7885318809926345e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280288804 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112893826 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500877157462196 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6914652722212 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚517852093251072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚7264.528521837 _step‚496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510911511745024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36881496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563610781792768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280262458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510911493742080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6940599508992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76012328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36880923 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚517853021225472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570865478893568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7254433892864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517852093251072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6941509480448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76012330å Q§î ûO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510911511745024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36881496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563610781792768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280262458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510911493742080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6940599508992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76012328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36880923 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚517853021225472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570865478893568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7254433892864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517852093251072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6941509480448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76012330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 % _timestamp‚1.7885318809926345e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280288804 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112893826 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500877157462196 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6914652722212 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚517852093251072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚7264.528521837 _step‚496ðzw2- _wandbruntime‚7265  _runtime‚7265ÁßR?'‘€N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255602314752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941720969216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36887079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚579 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36886500 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570956860423680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚563701007497216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916677970702 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112911474 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚563701031437312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570956609811968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126873894 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280333821 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76024395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚6049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112910893 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500958217344590 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126873892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517935761784832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚7265.542835868 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153433557 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126873892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510993112827904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494041539373888 _step‚497 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942630953984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76024393 % _timestamp‚1.7885318820067253e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494041522334528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0$âÈð '’…N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510993112827904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494041539373888 _step‚497 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942630953984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76024393 % _timestamp‚1.7885318820067253e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494041522334528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255602314752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941720969216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36887079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚579 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36886500 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570956860423680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚563701007497216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916677970702 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112911474 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚563701031437312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570956609811968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126873894 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280333821 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76024395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚6049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112910893 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500958217344590 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126873892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517935761784832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚7265.542835868 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153433557 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126873892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ñû Ž=2“- _wandbruntime‚7265  _runtime‚72653_VZ”Ô 0loss/FRt4/assignment_loss‚2.8810888595051236 1loss/SRqqt1/detection_loss‚2.7293186122956485 1loss/SRqqt3/detection_loss‚0.9627039592014626 /loss/FBt2/detection_loss‚3.0250811353325844 .loss/FBt4/detection_loss‚2.133667508761088 2loss/SRqqt2/assignment_loss‚1.6276825001049373 1loss/FBt3/assignment_loss‚0.06762973003613183 .loss/FRt3/detection_loss‚1.518773841396184 0loss/SRqqt2/detection_loss‚2.228169394450055 trainer/global_step‚6049 1loss/FBt2/assignment_loss‚0.07814994188083801 %loss/total_loss‚43.83726064965019 % _timestamp‚1.7885318820093753e+09 _runtime‚7265.544648326 /loss/FRt1/assignment_loss‚4.168813713218855 1loss/FBt1/assignment_loss‚0.09845965571649787 .loss/FRt4/detection_loss‚2.332617958303955 /loss/FBt3/detection_loss‚0.6244102832935159 1loss/FBt4/assignment_loss‚0.08726847493122136 /loss/FRt1/detection_loss‚3.2305589758831523 1loss/SRqqt4/detection_loss‚2.2212452869261465 /loss/FRt2/assignment_loss‚3.095018873612086 0loss/FRt3/assignment_loss‚1.2109906698616457 2loss/SRqqt3/assignment_loss‚0.8310456419033775 /loss/FBt1/detection_loss‚2.6202945245636835 /loss/FRt2/detection_loss‚2.6994563973512604 _step‚498 /loss/SRqqt1/assignment_loss‚1.9772175780457 2loss/SRqqt4/assignment_loss‚1.3875971330750374 epoch‚5Oô—f_•Ù /loss/FRt2/assignment_loss‚3.095018873612086 0loss/FRt3/assignment_loss‚1.2109906698616457 2loss/SRqqt3/assignment_loss‚0.8310456419033775 /loss/FBt1/detection_loss‚2.6202945245636835 /loss/FRt2/detection_loss‚2.6994563973512604 _step‚498 /loss/SRqqt1/assignment_loss‚1.9772175780457 2loss/SRqqt4/assignment_loss‚1.3875971330750374 epoch‚5 0loss/FRt4/assignment_loss‚2.8810888595051236 1loss/SRqqt1/detection_loss‚2.7293186122956485 1loss/SRqqt3/detection_loss‚0.9627039592014626 /loss/FBt2/detection_loss‚3.0250811353325844 .loss/FBt4/detection_loss‚2.133667508761088 2loss/SRqqt2/assignment_loss‚1.6276825001049373 1loss/FBt3/assignment_loss‚0.06762973003613183 .loss/FRt3/detection_loss‚1.518773841396184 0loss/SRqqt2/detection_loss‚2.228169394450055 trainer/global_step‚6049 1loss/FBt2/assignment_loss‚0.07814994188083801 %loss/total_loss‚43.83726064965019 % _timestamp‚1.7885318820093753e+09 _runtime‚7265.544648326 /loss/FRt1/assignment_loss‚4.168813713218855 1loss/FBt1/assignment_loss‚0.09845965571649787 .loss/FRt4/detection_loss‚2.332617958303955 /loss/FBt3/detection_loss‚0.6244102832935159 1loss/FBt4/assignment_loss‚0.08726847493122136 /loss/FRt1/detection_loss‚3.2305589758831523 1loss/SRqqt4/detection_loss‚2.2212452869261465ò?W““–‚(:ˆ ³©ëÔü·‚á gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n.Çq—‚(:æ ³©ëԦΗá disk.sda1.in‚0) disk./.usagePercent‚75.70097333708551! disk.sda2.out‚1263.2255859375 disk.md0.out‚ 1234.0703125 disk.dm-0.out‚114642.4921875 disk.sda.in‚128.625& proc.memory.availableMB‚ 1975.21875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1.52734375! disk.nvme0n1.out‚ 89219.015625 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 25423.4765625 disk.md0p1.out‚ 1234.0703125 disk.rbd0.in‚0 disk.rbd2.out‚ 2.3408203125 disk.dm-0.in‚ 16.9140625 disk.sda.out‚1263.2255859375 network.sent‚ 1.5560552e+07 disk.md0p1.in‚ 199.5859375) proc.memory.percent‚18.21458339691162 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 cpu‚25.237880540411645 disk.sdb2.in‚ 70.9609375% disk./.usageGB‚2707.4334831237793 disk.nvme1n1.in‚ 15.38671875 disk.sdb.out‚1263.2255859375! disk.sdb2.out‚1263.2255859375 network.recv‚ 2.430548e+06 disk.rbd0.out‚ 2.12841796875 disk.sdb1.in‚0# proc.memory.rssMB‚ 2984.27734375$ memory_percent‚87.94422149658203¹Uå“’˜‚(:‡ ©ëÔêšœà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î@:rÜ™‚(:ï ©ëÔóœôà disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.nvme0n1.in‚ 1.52734375 disk.md0.out‚ 1235.37109375 disk.sdb.in‚ 70.9609375! disk.sda.out‚1264.57470703125 network.sent‚ 901ž1.5669727e+07 disk.sdb1.in‚0! disk.dm-0.out‚114872.05859375 disk.sda.in‚128.625# proc.memory.rssMB‚ 2984.27734375 disk.md0p1.out‚ 1235.37109375 disk.sdb1.out‚0! disk.sdb.out‚1264.57470703125 disk.sda1.out‚0 disk.md0p1.in‚ 199.5859375" disk.sda2.out‚1264.57470703125) proc.memory.availableMB‚ 1975.21484375 cpu‚25.237451307144628 proc.cpu.threads‚17) disk./.usagePercent‚75.70097333708551# disk.nvme1n1.out‚25495.72265625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.3408203125" disk.nvme0n1.out‚ 89376.3359375" disk.sdb2.out‚1264.57470703125% disk./.usageGB‚2707.4334831237793 disk.rbd0.out‚ 2.19140625 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375$ memory_percent‚87.94424533843994 disk.dm-0.in‚ 16.9140625) proc.memory.percent‚18.21458339691162 network.recv‚ 2.435164e+06 disk.nvme1n1.in‚ 15.38671875 disk.rbd0.in‚0 ¦”š‚(:‰ Ñ©ëÔÝ›ùÞ gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.995' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.74875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”OÍt›‚(:é Ñ©ëÔ¨ðñà disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375$ memory_percent‚87.94584274291992 cpu‚25.236999649216916" disk.sda2.out‚1266.76611328125 disk.nvme1n1.in‚ 15.38671875) proc.memory.percent‚18.21458339691162 network.recv‚ 2.440307e+06! disk.nvme0n1.out‚ 89585.984375 disk.dm-0.in‚ 16.9140625! disk.sdb.out‚1266.76611328125 disk.md0p1.out‚ 1237.4921875 disk.nvme0n1.in‚ 1.52734375 disk.md0.out‚ 1237.4921875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.40380859375 network.sent‚ 1.5674205e+07% disk./.usageGB‚2707.4334831237793 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚115205.1171875) disk./.usagePercent‚75.70097333708551 disk.sda1.out‚0 disk.sda.in‚128.625! disk.sda.out‚1266.76611328125' proc.memory.availableMB‚ 1974.953125 proc.cpu.threads‚17 disk.sda1.in‚0# proc.memory.rssMB‚ 2984.27734375 disk.rbd0.in‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1266.76611328125" disk.nvme1n1.out‚ 25619.1328125 disk.rbd0.out‚ 2.19140625£ßæl’œ‚(:‡ à©ëÔ͸ÓÞ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.552' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ì( x‚(:í à©ëÔƒœÇà! disk.dm-0.out‚115547.66796875! disk.sda.out‚1268.07763671875$ memory_percent‚87.94584274291992) disk./.usagePercent‚75.68616202895296 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 2.25439453125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1.52734375 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 2.40380859375 cpu‚25.236586045030407 disk.dm-0.in‚ 16.9140625 disk.nvme1n1.in‚ 15.38671875% disk./.usageGB‚2706.9037590026855 disk.md0.in‚ 199.5859375" disk.nvme1n1.out‚ 25699.3515625 disk.rbd0.in‚0# disk.nvme0n1.out‚89848.31640625' proc.memory.availableMB‚ 1974.953125 network.sent‚ 1.5683111e+07" disk.sdb2.out‚1268.07763671875 disk.md0p1.out‚ 1238.734375 disk.md0.out‚ 1238.734375 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0# proc.memory.rssMB‚ 2984.27734375) proc.memory.percent‚18.21458339691162 network.recv‚ 2.445347e+06! disk.sdb.out‚1268.07763671875 disk.sda2.in‚128.625" disk.sda2.out‚1268.07763671875¡]x2ž- _wandbruntime‚7320  _runtime‚7320³8ì·ÉŸÃ _step‚499 ' lr-AdamW/pg1‚0.00042381197950295494 ' lr-AdamW/pg2‚0.00042381197950295494 trainer/global_step‚6099 % _timestamp‚1.7885319367683778e+09 _runtime‚7320.303860267oTÎ È _step‚499 ' lr-AdamW/pg1‚0.00042381197950295494 ' lr-AdamW/pg2‚0.00042381197950295494 trainer/global_step‚6099 % _timestamp‚1.7885319367683778e+09 _runtime‚7320.303860267ó‚Üa±2¡- _wandbruntime‚7320  _runtime‚7320#űøü'¢öO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6971437952324 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282515360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997596905472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514992448033280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113780736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113780134 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7312726760960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7312489590272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154653628 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6997596905472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76615580 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚504883914444504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127861734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575434817716224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127861734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76615578 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37165156 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514992466036224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521989134967296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚500îiü(£ûO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514992448033280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127861732 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154653628 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521990062941696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6996686934016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚504884839458868 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚497913401506544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6970529977320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514992466036224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127861732 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚6099 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514992466036224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521989134967296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚500 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6971437952324 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282515360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997596905472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514992448033280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113780736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113780134 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7312726760960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7312489590272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154653628 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6997596905472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76615580 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚504883914444504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127861734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575434817716224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127861734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76615578 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37165156 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113797830 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37170187 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚501 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575525952015872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7321.269234331 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚504964049312326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6972555225810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154678459 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚6099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282560351 \w{²; PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113798445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568212292719616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515074067119104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515074067119104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127881894 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154704829 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522072803501056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522072803501056ƒî¦G '¦…N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚504964974326898 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127881892 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113797830 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37170187 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚501 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575525952015872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7321.269234331 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚504964049312326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6972555225810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154678459 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚6099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282560351 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113798445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568212292719616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515074067119104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515074067119104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127881894 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154704829 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522072803501056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522072803501056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76627645 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127881892 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7313885967872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6971647250598 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6998718379008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76627643 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575526202627584õº9362§- _wandbruntime‚7321  _runtime‚7321~Ê+B]¨× 1loss/SRqqt3/detection_loss‚0.9259676785791764 .loss/FBt4/detection_loss‚3.275493288040161 2loss/SRqqt4/assignment_loss‚1.5130930737975765 0loss/FBt3/assignment_loss‚0.0577283525291581 1loss/SRqqt1/detection_loss‚2.6688770191235975 /loss/FBt3/detection_loss‚0.6442813070703156 epoch‚5 &loss/total_loss‚42.877630376741315 _step‚502 0loss/FRt2/assignment_loss‚3.5181449362725923 1loss/FBt1/assignment_loss‚0.10939670289155856 /loss/FRt3/detection_loss‚1.4142654659226537 1loss/SRqqt1/assignment_loss‚1.836596811359579 trainer/global_step‚6099 _runtime‚ 7321.27107771 2loss/SRqqt2/assignment_loss‚1.3784660256427268 /loss/FRt2/detection_loss‚2.2321534546825923 .loss/FBt1/detection_loss‚3.308559913188219 % _timestamp‚1.7885319377358055e+09 0loss/FRt3/assignment_loss‚1.2515014269772697 2loss/SRqqt3/assignment_loss‚1.0287798127406333 1loss/FBt2/assignment_loss‚0.14257608820436055 2loss/FBt4/assignment_loss‚0.010442058221542538 1loss/SRqqt2/detection_loss‚1.8861018859821816 /loss/SRqqt4/detection_loss‚2.07812760682667 /loss/FBt2/detection_loss‚1.7373850538450129 0loss/FRt1/assignment_loss‚2.8035537292217385 0loss/FRt4/assignment_loss‚3.4742077619601517 .loss/FRt1/detection_loss‚2.949430500638896 /loss/FRt4/detection_loss‚2.6325004230229516<45|b©Ü epoch‚5 &loss/total_loss‚42.877630376741315 _step‚502 0loss/FRt2/assignment_loss‚3.5181449362725923 1loss/FBt1/assignment_loss‚0.10939670289155856 /loss/FRt3/detection_loss‚1.4142654659226537 1loss/SRqqt1/assignment_loss‚1.836596811359579 trainer/global_step‚6099 _runtime‚ 7321.27107771 2loss/SRqqt2/assignment_loss‚1.3784660256427268 /loss/FRt2/detection_loss‚2.2321534546825923 .loss/FBt1/detection_loss‚3.308559913188219 % _timestamp‚1.7885319377358055e+09 0loss/FRt3/assignment_loss‚1.2515014269772697 2loss/SRqqt3/assignment_loss‚1.0287798127406333 1loss/FBt2/assignment_loss‚0.14257608820436055 2loss/FBt4/assignment_loss‚0.010442058221542538 1loss/SRqqt2/detection_loss‚1.8861018859821816 /loss/SRqqt4/detection_loss‚2.07812760682667 /loss/FBt2/detection_loss‚1.7373850538450129 0loss/FRt1/assignment_loss‚2.8035537292217385 0loss/FRt4/assignment_loss‚3.4742077619601517 .loss/FRt1/detection_loss‚2.949430500638896 /loss/FRt4/detection_loss‚2.6325004230229516 1loss/SRqqt3/detection_loss‚0.9259676785791764 .loss/FBt4/detection_loss‚3.275493288040161 2loss/SRqqt4/assignment_loss‚1.5130930737975765 0loss/FBt3/assignment_loss‚0.0577283525291581 1loss/SRqqt1/detection_loss‚2.6688770191235975 /loss/FBt3/detection_loss‚0.6442813070703156öPØ%J”ª‚(:‰ ï©ëÔöº®à gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶6âx«‚(:í ï©ëÔÕ¯˜ã disk.nvme0n1.in‚ 1.52734375 disk.md0.out‚ 1240.109375 proc.cpu.threads‚17 network.sent‚ 1.5699075e+07% disk./.usageGB‚2706.9034423828125! disk.sda.out‚1269.51513671875 disk.nvme1n1.in‚ 15.38671875 disk.sda1.out‚0" disk.sda2.out‚1269.51513671875 disk.md0.in‚ 199.5859375! disk.sdb.out‚1269.51513671875 network.recv‚ 2.449334e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 90005.5546875" disk.sdb2.out‚1269.51513671875) proc.memory.availableMB‚ 1975.16796875) proc.memory.percent‚18.21458339691162 cpu‚25.236186881369303 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0p1.out‚ 1240.109375 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.dm-0.in‚ 16.9140625! disk.dm-0.out‚115759.96484375# proc.memory.rssMB‚ 2984.27734375# disk.nvme1n1.out‚25754.41015625 disk.sda1.in‚0$ memory_percent‚87.94453144073486) disk./.usagePercent‚75.68615317612878 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.25439453125 disk.rbd2.out‚ 2.466796875é¸"’¬‚(:‡ þ©ëÔ¹Þ¦à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼µ±áw­‚(:ì þ©ëÔª–™á disk.md0.in‚ 199.5859375! disk.sdb.out‚1270.90185546875# proc.memory.rssMB‚ 2984.27734375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1.52734375 disk.md0.out‚ 1241.44921875 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1270.90185546875 network.recv‚ 2.454761e+06 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚90091.92578125 disk.dm-0.out‚ 115847.234375# disk.nvme1n1.out‚25755.30859375# memory_percent‚87.9446029663086& proc.memory.availableMB‚ 1975.15625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.25439453125 disk.sda1.out‚0" disk.sda2.out‚1270.90185546875) proc.memory.percent‚18.21458339691162 network.sent‚ 1.5795889e+07 disk.rbd2.in‚0! disk.sda.out‚1270.90185546875% disk./.usageGB‚2706.9310150146484 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 15.38671875 disk.sda.in‚128.625 disk.rbd2.out‚ 2.466796875) disk./.usagePercent‚75.68692411845497 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1241.44921875 disk.sda2.in‚128.625 disk.dm-0.in‚ 16.9140625 cpu‚25.235754485241113Z÷‹“®‚(:ˆ ªëÔéÄîÞ gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óáýty¯‚(:î ªëÔ‡¢êá network.recv‚ 2.466315e+06 disk.dm-0.out‚ 116112.171875 disk.sda2.in‚128.625# disk.nvme1n1.out‚25755.45703125 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1243.0859375 disk.rbd2.in‚0) disk./.usagePercent‚75.68699846084586% disk./.usageGB‚2706.9336738586426 disk.nvme0n1.in‚ 1.52734375 disk.md0.out‚ 1243.0859375 disk.nvme1n1.in‚ 15.38671875 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1272.59326171875 disk.sda1.out‚0) proc.memory.availableMB‚ 1975.16796875 disk.rbd2.out‚ 2.466796875 proc.cpu.threads‚17 network.sent‚ 1.5805998e+07 disk.dm-0.in‚ 16.9140625! disk.sda.out‚1272.59326171875 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0) proc.memory.percent‚18.21458339691162 disk.rbd0.out‚ 2.25439453125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 cpu‚25.235339213687798" disk.sda2.out‚1272.59326171875! disk.sdb.out‚1272.59326171875 disk.sda.in‚128.625 disk.rbd0.in‚0# disk.nvme0n1.out‚90356.71484375$ memory_percent‚87.94453144073486# proc.memory.rssMB‚ 2984.27734375x2 2°- _wandbruntime‚7375  _runtime‚7375Åø­’ɱà _step‚503 ' lr-AdamW/pg1‚0.00041859052689068927 ' lr-AdamW/pg2‚0.00041859052689068927 trainer/global_step‚6149 % _timestamp‚1.7885319922731822e+09 _runtime‚7375.808641724Á6ìUÎ²È _runtime‚7375.808641724 _step‚503 ' lr-AdamW/pg1‚0.00041859052689068927 ' lr-AdamW/pg2‚0.00041859052689068927 trainer/global_step‚6149 % _timestamp‚1.7885319922731822e+09÷h«G2³- _wandbruntime‚7375  _runtime‚7375ºÈÜâü'´öO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155898530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885319922741249e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77218830 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052774359040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114666878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114666275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526127104657920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7375.809723649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519073420327424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519073402324480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526126176683520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572633352237568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128869734 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371018042880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7052774359040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77218828 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526127104657920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128869734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580004151984128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155924874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572633376177664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155924874 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37448048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚504<{H:µûO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128869734 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371018042880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7052774359040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77218828 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526127104657920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128869734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580004151984128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155924874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572633376177664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155924874 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37448048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155898530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885319922741249e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77218830 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052774359040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114666878 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114666275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526127104657920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7375.809723649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519073420327424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519073402324480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526126176683520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572633352237568 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114684046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37453153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚519155021410304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114684666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7054805804032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155923361 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7028432480918 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284839625 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚526208917229568 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128889894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155949731 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _runtime‚7376.870107723 _step‚505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580095807234560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7027524505706 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7054805804032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128889894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155949731 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7371982875136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128889892 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77230895 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚572723577942016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7053895819264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508970806294634 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192i—rŸ '¸…N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128889894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155949731 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7371982875136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128889892 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77230895 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚572723577942016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7053895819264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508970806294634 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114684046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37453153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚519155021410304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114684666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7054805804032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155923361 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7028432480918 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284839625 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚526208917229568 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128889894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155949731 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _runtime‚7376.870107723 _step‚505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580095807234560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7027524505706 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7054805804032ùa&-2¹- _wandbruntime‚7376  _runtime‚7376Ö »[ºÕ 2loss/SRqqt1/assignment_loss‚1.7030231466999761 1loss/SRqqt4/assignment_loss‚1.320955381431478 /loss/FRt2/assignment_loss‚2.643093773296901 1loss/SRqqt1/detection_loss‚2.2892747742158397 0loss/SRqqt2/detection_loss‚2.114890308039529 1loss/SRqqt4/detection_loss‚2.0451527504210776 %loss/total_loss‚38.67492922612734 _runtime‚7376.871087127 _step‚506 2loss/SRqqt3/assignment_loss‚0.8538756664770032 1loss/FBt2/assignment_loss‚0.10664635555370619 1loss/FBt4/assignment_loss‚0.06800487030361235 .loss/FRt1/detection_loss‚2.883504637650081 /loss/FRt3/detection_loss‚1.3183961606942691 2loss/SRqqt2/assignment_loss‚1.4468638473323414 1loss/FBt3/assignment_loss‚0.04166574498720541 /loss/FRt2/detection_loss‚2.7221535698408803 /loss/FBt2/detection_loss‚2.0528029909259393 % _timestamp‚1.7885319933357737e+09 /loss/FRt1/assignment_loss‚2.480285108089447 0loss/FRt3/assignment_loss‚0.9510859461931082 .loss/FRt4/detection_loss‚2.058195952905549 .loss/FBt4/detection_loss‚2.255600517286974 epoch‚5 trainer/global_step‚6149 1loss/FBt1/assignment_loss‚0.30234403549660355 1loss/SRqqt3/detection_loss‚0.7751655355457542 /loss/FBt1/detection_loss‚2.6216136182055756 0loss/FRt4/assignment_loss‚3.0204808897442286 .loss/FBt3/detection_loss‚0.599853644790259^G9`»Ú /loss/FBt2/detection_loss‚2.0528029909259393 % _timestamp‚1.7885319933357737e+09 /loss/FRt1/assignment_loss‚2.480285108089447 0loss/FRt3/assignment_loss‚0.9510859461931082 .loss/FRt4/detection_loss‚2.058195952905549 .loss/FBt4/detection_loss‚2.255600517286974 epoch‚5 trainer/global_step‚6149 1loss/FBt1/assignment_loss‚0.30234403549660355 1loss/SRqqt3/detection_loss‚0.7751655355457542 /loss/FBt1/detection_loss‚2.6216136182055756 0loss/FRt4/assignment_loss‚3.0204808897442286 .loss/FBt3/detection_loss‚0.599853644790259 2loss/SRqqt1/assignment_loss‚1.7030231466999761 1loss/SRqqt4/assignment_loss‚1.320955381431478 /loss/FRt2/assignment_loss‚2.643093773296901 1loss/SRqqt1/detection_loss‚2.2892747742158397 0loss/SRqqt2/detection_loss‚2.114890308039529 1loss/SRqqt4/detection_loss‚2.0451527504210776 %loss/total_loss‚38.67492922612734 _runtime‚7376.871087127 _step‚506 2loss/SRqqt3/assignment_loss‚0.8538756664770032 1loss/FBt2/assignment_loss‚0.10664635555370619 1loss/FBt4/assignment_loss‚0.06800487030361235 .loss/FRt1/detection_loss‚2.883504637650081 /loss/FRt3/detection_loss‚1.3183961606942691 2loss/SRqqt2/assignment_loss‚1.4468638473323414 1loss/FBt3/assignment_loss‚0.04166574498720541 /loss/FRt2/detection_loss‚2.7221535698408803ú1X"“¼‚(:ˆ œªëÔį®à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å»b‘t½‚(:é œªëÔÙ‹Ÿã disk.sda1.in‚0! disk.sdb.out‚1274.28662109375 disk.nvme1n1.in‚ 15.38671875" disk.sdb2.out‚1274.28662109375! disk.sda.out‚1274.28662109375 network.recv‚ 2.477879e+06 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375 network.sent‚ 1.5826832e+07 disk.rbd0.out‚ 2.25439453125 disk.rbd2.out‚ 2.466796875 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.rbd2.in‚0# disk.nvme0n1.out‚90568.23828125) disk./.usagePercent‚75.69050087273914 disk.sda1.out‚0 disk.sda.in‚128.625$ memory_percent‚87.94634342193604# proc.memory.rssMB‚ 2984.27734375 cpu‚25.23495351016169! disk.dm-0.out‚116318.26171875$ disk./.usageGB‚2707.058937072754 disk.sdb1.out‚0" disk.sda2.out‚1274.28662109375 disk.md0.out‚ 1244.71875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1244.71875) proc.memory.availableMB‚ 1974.87109375 disk.nvme0n1.in‚ 1.52734375 disk.dm-0.in‚ 16.9140625" disk.nvme1n1.out‚ 25756.0234375) proc.memory.percent‚18.21458339691162 proc.cpu.threads‚17É6¼¾‚(:ˆ «ªëÔÅѼà gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.pom‡Y×werWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½!‚–z¿‚(:ï «ªëÔ¥Ìùá disk.sda1.out‚0 disk.nvme0n1.in‚ 1.52734375! disk.sdb.out‚1275.69384765625 disk.sda.in‚128.625) disk./.usagePercent‚75.69398685891046# disk.nvme1n1.out‚25756.25390625 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1246.07421875 network.sent‚ 1.5922771e+07 disk.sdb.in‚ 70.9609375# proc.memory.rssMB‚ 2984.27734375 cpu‚25.23452148068966 disk.nvme1n1.in‚ 15.38671875 disk.md0p1.out‚ 1246.07421875# disk.nvme0n1.out‚90813.11328125 disk.dm-0.in‚ 16.9140625! disk.sda.out‚1275.69384765625 proc.cpu.threads‚17 network.recv‚ 2.482355e+06 disk.md0p1.in‚ 199.5859375$ memory_percent‚87.94722557067871( proc.memory.availableMB‚ 1974.7265625) proc.memory.percent‚18.21458339691162" disk.sdb2.out‚1275.69384765625% disk./.usageGB‚2707.1836128234863 disk.rbd0.in‚0 disk.rbd0.out‚ 2.25439453125 disk.sda1.in‚0 disk.md0.in‚ 199.5859375 disk.dm-0.out‚116569.3671875 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625" disk.sda2.out‚1275.69384765625Èðy,“À‚(:ˆ ºªëÔø²æà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ÏM|Á‚(:ñ ºªëÔ„×»ä" disk.sda2.out‚1277.44384765625! disk.sdb.out‚1277.44384765625) proc.memory.percent‚18.21458339691162" disk.sdb2.out‚1277.44384765625 disk.nvme1n1.in‚ 15.38671875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2984.27734375 proc.cpu.threads‚17 disk.md0p1.out‚ 1247.75390625 disk.rbd0.out‚ 2.25439453125 disk.rbd2.out‚ 2.466796875# disk.nvme0n1.out‚91023.41796875) proc.memory.availableMB‚ 1974.33984375) disk./.usagePercent‚75.70100544190568 disk.nvme0n1.in‚ 1.52734375 network.sent‚ 1.5929272e+07 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚25756.52734375 disk.md0.in‚ 199.5859375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625 disk.dm-0.out‚116779.9453125$ memory_percent‚87.94958591461182 network.recv‚ 2.492309e+06 disk.dm-0.in‚ 16.9140625 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.md0.out‚ 1247.75390625 cpu‚25.234030579550495% disk./.usageGB‚2707.4346313476562 disk.sda.in‚128.625! disk.sda.out‚1277.44384765625 disk.rbd0.in‚05h#’‚(:‡ ɪëÔ’’°à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q—®rÂ(:ç ɪëÔÒó–á proc.cpu.threads‚17( disk./.usagePercent‚75.7009769635436 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625! disk.sdb2.out‚1279.2548828125% disk./.usageGB‚2707.4336128234863 disk.sda.in‚128.625 disk.rbd0.out‚ 2.25439453125 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sda2.out‚1279.2548828125 disk.md0.out‚ 1249.4921875 disk.sdb.in‚ 70.9609375 disk.sda.out‚1279.2548828125 disk.sdb.out‚1279.2548828125# proc.memory.rssMB‚ 2984.27734375 cpu‚25.23351518721673 disk.nvme1n1.in‚ 15.38671875 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1249.4921875 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 network.recv‚ 2.504424e+06# disk.nvme1n1.out‚25756.67578125# disk.nvme0n1.out‚91344.01953125 disk.dm-0.out‚117100.6953125$ memory_percent‚91.96538925170898 disk.rbd2.out‚ 2.466796875' proc.memory.availableMB‚ 1316.390625 network.sent‚ 1.5939257e+07 disk.nvme0n1.in‚ 1.52734375 disk.dm-0.in‚ 16.9140625) proc.memory.percent‚18.21458339691162( ?I2Ä- _wandbruntime‚7432  _runtime‚7432wûø[ÉÅà _step‚507 ' lr-AdamW/pg1‚0.00041335809359688226 ' lr-AdamW/pg2‚0.00041335809359688226 trainer/global_step‚6199 % _timestamp‚1.7885320492627244e+09 _runtime‚7432.7982998395/O¼ÎÆÈ _step‚507 ' lr-AdamW/pg1‚0.00041335809359688226 ' lr-AdamW/pg2‚0.00041335809359688226 trainer/global_step‚6199 % _timestamp‚1.7885320492627244e+09 _runtime‚7432.798299839û\z¦2Ç- _wandbruntime‚7432  _runtime‚7432VïIÀü'ÈöO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7083192462540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115555578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7429011542016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77822078 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157143432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7108861784064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129877734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚7432.799393654 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287047510 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7082284487536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚6199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530263218399744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37733498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512897428409120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129877732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77822080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚505815160960944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115554957 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530263218399744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530264146374144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577144637459968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523154374618624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360êf°Ñ(ÉûO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚505815143921584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523154356615680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚584573899627008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577144661400064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287021164 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7108861784064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129877734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚7432.799393654 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287047510 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7082284487536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚6199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530263218399744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37733498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512897428409120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129877732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77822080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚505815160960944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115554957 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530263218399744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530264146374144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577144637459968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523154374618624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7083192462540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115555578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7429011542016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77822078 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157143432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7433.800666388 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.alloca›WØö ted‚ 287092527 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115572645 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7430184987648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚505894161516128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157168263 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚509 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157168263 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7109983244288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37738502 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7110893229056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584665296666624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚530345958945792ï¥t­ 'Ì…N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7433.800666388 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287092527 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115572645 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7430184987648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚505894161516128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157168263 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚509 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157168263 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7109983244288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37738502 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7110893229056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584665296666624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚530345958945792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287066155 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129897894 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577234887104512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚505894178555488ýõDàu2Í- _wandbruntime‚7433  _runtime‚7433«ËÆ6\ÎÖ 1loss/FBt1/assignment_loss‚0.14010629022062915 0loss/FBt4/assignment_loss‚0.2543915654578844 1loss/SRqqt1/detection_loss‚2.3149588083227477 /loss/FRt2/detection_loss‚2.2674337754163845 /loss/FRt3/detection_loss‚1.5539449456846341 0loss/FRt2/assignment_loss‚3.2533167943358423 1loss/FBt2/assignment_loss‚0.15285070573929257 1loss/SRqqt2/detection_loss‚2.1510171284278234 trainer/global_step‚6199 _step‚510 0loss/FRt1/assignment_loss‚3.2785244357018244 2loss/SRqqt1/assignment_loss‚1.5526599739160802 2loss/SRqqt3/assignment_loss‚0.9908236216011429 %loss/total_loss‚41.56734270910022 _runtime‚7433.801759423 /loss/FBt3/detection_loss‚0.5365141401865653 /loss/FRt4/detection_loss‚2.2366374882558984 1loss/SRqqt3/detection_loss‚0.8270838535068059 1loss/SRqqt2/assignment_loss‚2.203608394662539 1loss/FBt3/assignment_loss‚0.04856415149191337 epoch‚5 $ _timestamp‚1.788532050266505e+09 0loss/FRt3/assignment_loss‚1.1720609664916992 0loss/SRqqt4/detection_loss‚2.004369863649694 /loss/FBt2/detection_loss‚2.1842342146805356 .loss/FBt4/detection_loss‚2.204214329520861 .loss/FBt1/detection_loss‚2.185243807733059 .loss/FRt1/detection_loss‚3.332711094901675 0loss/FRt4/assignment_loss‚3.2802284823523626 2loss/SRqqt4/assignment_loss‚1.4418438768423185ÅX aÏÛ 0loss/FRt2/assignment_loss‚3.2533167943358423 1loss/FBt2/assignment_loss‚0.15285070573929257 1loss/SRqqt2/detection_loss‚2.1510171284278234 trainer/global_step‚6199 _step‚510 0loss/FRt1/assignment_loss‚3.2785244357018244 2loss/SRqqt1/assignment_loss‚1.5526599739160802 2loss/SRqqt3/assignment_loss‚0.9908236216011429 %loss/total_loss‚41.56734270910022 _runtime‚7433.801759423 /loss/FBt3/detection_loss‚0.5365141401865653 /loss/FRt4/detection_loss‚2.2366374882558984 1loss/SRqqt3/detection_loss‚0.8270838535068059 1loss/SRqqt2/assignment_loss‚2.203608394662539 1loss/FBt3/assignment_loss‚0.04856415149191337 epoch‚5 $ _timestamp‚1.788532050266505e+09 0loss/FRt3/assignment_loss‚1.1720609664916992 0loss/SRqqt4/detection_loss‚2.004369863649694 /loss/FBt2/detection_loss‚2.1842342146805356 .loss/FBt4/detection_loss‚2.204214329520861 .loss/FBt1/detection_loss‚2.185243807733059 .loss/FRt1/detection_loss‚3.332711094901675 0loss/FRt4/assignment_loss‚3.2802284823523626 2loss/SRqqt4/assignment_loss‚1.4418438768423185 1loss/FBt1/assignment_loss‚0.14010629022062915 0loss/FBt4/assignment_loss‚0.2543915654578844 1loss/SRqqt1/detection_loss‚2.3149588083227477 /loss/FRt2/detection_loss‚2.2674337754163845 /loss/FRt3/detection_loss‚1.5539449456846341þ&”‰”Ђ(:‰ تëÔÁâ˜á gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²JCgÑ‚(:Ü تëÔ®ÊÝá disk.sda1.in‚0 disk.md0p1.out‚ 1250.8828125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 1.52734375 disk.sdb2.out‚ 1280.7109375 disk.sda.in‚128.625 network.sent‚ 1.5964736e+07 disk.md0.out‚ 1250.8828125 disk.dm-0.in‚ 16.9140625 disk.sdb2.in‚ 70.9609375# proc.memory.rssMB‚ 2984.27734375$ disk./.usageGB‚2707.433666229248 disk.nvme0n1.out‚ 91664.03125 disk.md0.in‚ 199.5859375! disk.dm-0.out‚117420.94140625$ memory_percent‚91.96550846099854) proc.memory.availableMB‚ 1316.37109375 cpu‚25.233108975006697 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sda2.out‚ 1280.7109375 disk.sda.out‚ 1280.7109375# disk.nvme1n1.out‚25756.91015625) disk./.usagePercent‚75.70097845679105 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.rbd0.out‚ 2.25439453125 disk.nvme1n1.in‚ 15.38671875) proc.memory.percent‚18.21458339691162 disk.sdb1.in‚0 disk.sdb.out‚ 1280.7109375 network.recv‚ 2.521129e+06ã,q/“Ò‚(:ˆ çªëÔªØÊà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ì‘–yÓ‚(:î çªëÔ·¸³á disk.rbd0.out‚ 2.25439453125 network.sent‚ 1.6058794e+07 disk.rbd2.in‚0 disk.md0.out‚ 1252.83203125! disk.sda.out‚1282.72021484375 disk.nvme1n1.in‚ 15.38671875* proc.memory.percent‚18.214964866638184 network.recv‚ 2.526394e+06 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875" disk.sda2.out‚1282.72021484375 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1282.72021484375# disk.nvme1n1.out‚25757.10546875$ memory_percent‚91.96557998657227 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 2984.33984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1.52734375# disk.nvme0n1.out‚91829.10546875 disk.dm-0.out‚117586.2109375 disk.sda2.in‚128.625 disk.dm-0.in‚ 16.9140625 disk.md0p1.out‚ 1252.83203125' proc.memory.availableMB‚ 1316.359375% disk./.usageGB‚2707.4335174560547 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 cpu‚25.23274406989839( disk./.usagePercent‚75.7009742970303" disk.sdb2.out‚1282.72021484375 disk.md0p1.in‚ 199.5859375!˜^è‘Ô‚(:† öªëÔÜ §à gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì,_oÕ‚(:ä öªëÔæÙàà* proc.memory.percent‚18.214964866638184 disk.dm-0.in‚20.53125 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.19140625 disk.md0.in‚ 199.5859375) proc.memory.availableMB‚ 1316.11328125 disk.rbd0.in‚0" disk.sdb2.out‚1283.99072265625 disk.nvme1n1.out‚ 25873.96875 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375! disk.sda.out‚1283.99072265625 network.recv‚ 2.530354e+06) disk./.usagePercent‚75.68965782789326 network.sent‚ 1.6064532e+07 disk.md0p1.out‚ 1254.04296875 disk.rbd0.out‚ 2.25439453125# memory_percent‚91.9670820236206# proc.memory.rssMB‚ 2984.33984375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda2.in‚128.625" disk.sda2.out‚1283.99072265625 disk.md0.out‚ 1254.04296875 disk.sdb1.out‚0 cpu‚25.23228275009902% disk./.usageGB‚2707.0287857055664 disk.nvme0n1.out‚ 91991.375 disk.dm-0.out‚ 117861.34375! disk.sdb.out‚1283.99072265625 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 16.33984375—- M”Ö‚(:‰ …«ëÔ±®•à gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ÿicׂ(:Ø …«ëÔþå§á% disk./.usageGB‚2706.9726943969727 disk.sda.out‚ 1285.58984375 disk.nvme1n1.in‚ 19.140625 disk.sda2.out‚ 1285.58984375 disk.dm-0.in‚ 24.26171875' proc.memory.availableMB‚ 1316.546875* proc.memory.percent‚18.214964866638184 disk.md0p1.out‚ 1255.59375 disk.rbd0.out‚ 2.25439453125 disk.sda1.in‚0 disk.sda2.in‚128.625 proc.cpu.threads‚17 network.recv‚ 2.541572e+06 disk.sda.in‚128.625 disk.sda1.out‚0 disk.nvme0n1.out‚ 92199.59375 disk.md0.out‚ 1255.59375 network.sent‚ 1.6076257e+07) disk./.usagePercent‚75.68808949142894 disk.sdb.out‚ 1285.58984375 disk.rbd0.in‚0 disk.dm-0.out‚118190.1484375$ memory_percent‚91.96443557739258 disk.nvme0n1.in‚ 5.12109375 disk.md0.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚ 1285.58984375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875" disk.nvme1n1.out‚ 25990.5546875 disk.sdb1.in‚0# proc.memory.rssMB‚ 2984.33984375 cpu‚25.23191473683719 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0åý?2Ø- _wandbruntime‚7490  _runtime‚7490Õ®DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130885732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7138161742644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚589143293795840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7487371113472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527235328909824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130885734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534401188090368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289300412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38019268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7139069717648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534400260115968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78425330 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158414678 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7487610381312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚6249 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7490.485037969 _step‚512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130885732 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78425328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7165859180544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7490.485037969 _step‚512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130885732 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78425328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7165859180544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130885732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7138161742644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚589143293795840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7487371113472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527235328909824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130885734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534401188090368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289300412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38019268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7139069717648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534400260115968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78425330 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0€Ý߸2Þ- _wandbruntime‚7491  _runtime‚7491‹>‘'ßÿM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7166070669312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7488544958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527316947995648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78437395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38024965 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚527316929992704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589234693345280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7488769532928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158413165 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130905892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7140186991134 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589234941859840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158439535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116461733 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158439535 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509845041243328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289319057 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581746148386816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534483928649728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527316929992704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7166980654080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534483000662016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78437393 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534483928649728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7166070669312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516985245273822 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289345429 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚6249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788532107929487e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130905894 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130905894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289319057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289345429»¦D 'à„N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78437395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38024965 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚527316929992704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589234693345280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7488769532928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158413165 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130905892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7140186991134 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589234941859840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158439535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116461733 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158439535 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509845041243328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289319057 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581746148386816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534483928649728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527316929992704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7166980654080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534483000662016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78437393 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534483928649728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7166070669312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516985245273822 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289345429 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚6249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788532107929487e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130905894 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130905894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289319057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289345429 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7166070669312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7488544958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527316947995648þÖãq2á- _wandbruntime‚7491  _runtime‚7491ØDktaâÛ 2loss/SRqqt1/assignment_loss‚1.3827817817883832 2loss/SRqqt2/assignment_loss‚1.4137749912780384 /loss/FRt2/detection_loss‚2.3746218050095416 1loss/SRqqt1/detection_loss‚2.5680716963750974 /loss/FBt1/detection_loss‚2.7102148234844208 .loss/FBt2/detection_loss‚1.725261055904886 trainer/global_step‚6249 0loss/FRt3/assignment_loss‚1.3547256886959076 /loss/FBt3/detection_loss‚0.6489678628961829 /loss/FBt4/detection_loss‚2.5869630947709084 %loss/total_loss‚41.61629601786285 % _timestamp‚1.7885321079321988e+09 0loss/FRt4/assignment_loss‚2.9455429708085408 1loss/SRqqt4/detection_loss‚2.4302154696649976 _runtime‚7491.467463356 _step‚514 2loss/SRqqt3/assignment_loss‚1.0089769808398117 1loss/FBt1/assignment_loss‚0.14985134839192446 2loss/FBt3/assignment_loss‚0.007677413084112798 2loss/SRqqt4/assignment_loss‚1.4481622629488509 0loss/FRt2/assignment_loss‚3.6016923478155425 0loss/FBt4/assignment_loss‚0.4318016911674931 /loss/FRt4/detection_loss‚2.6251028156498584 0loss/FRt1/assignment_loss‚2.8668985813856125 /loss/FRt3/detection_loss‚1.2754150383375786 epoch‚5 1loss/FBt2/assignment_loss‚0.25126925489837937 /loss/FRt1/detection_loss‚3.1624649514754615 0loss/SRqqt2/detection_loss‚1.891070615413577 1loss/SRqqt3/detection_loss‚0.7547714757777396ôE¿ ãà 0loss/FRt2/assignment_loss‚3.6016923478155425 0loss/FBt4/assignment_loss‚0.4318016911674931 /loss/FRt4/detection_loss‚2.6251028156498584 0loss/FRt1/assignment_loss‚2.8668985813856125 /loss/FRt3/detection_loss‚1.2754150383375786 epoch‚5 1loss/FBt2/assignment_loss‚0.25126925489837937 /loss/FRt1/detection_loss‚3.1624649514754615 0loss/SRqqt2/detection_loss‚1.891070615413577 1loss/SRqqt3/detection_loss‚0.7547714757777396 2loss/SRqqt1/assignment_loss‚1.3827817817883832 2loss/SRqqt2/assignment_loss‚1.4137749912780384 /loss/FRt2/detection_loss‚2.3746218050095416 1loss/SRqqt1/detection_loss‚2.5680716963750974 /loss/FBt1/detection_loss‚2.7102148234844208 .loss/FBt2/detection_loss‚1.725261055904886 trainer/global_step‚6249 0loss/FRt3/assignment_loss‚1.3547256886959076 /loss/FBt3/detection_loss‚0.6489678628961829 /loss/FBt4/detection_loss‚2.5869630947709084 %loss/total_loss‚41.61629601786285 % _timestamp‚1.7885321079321988e+09 0loss/FRt4/assignment_loss‚2.9455429708085408 1loss/SRqqt4/detection_loss‚2.4302154696649976 _runtime‚7491.467463356 _step‚5r’Ö14 2loss/SRqqt3/assignment_loss‚1.0089769808398117 1loss/FBt1/assignment_loss‚0.14985134839192446 2loss/FBt3/assignment_loss‚0.007677413084112798 2loss/SRqqt4/assignment_loss‚1.4481622629488509‚Äã^í“ä‚(:ˆ ”«ëÔƒ­à gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“:B?på‚(:å ”«ëÔ´Ìíá disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 24.26171875 disk.sdb2.in‚ 70.9609375( proc.memory.availableMB‚ 1316.2578125 disk.sdb1.in‚0 disk.sdb2.out‚1287.587890625% disk./.usageGB‚2706.9312286376953 disk.sda2.in‚128.625 disk.sda2.out‚1287.587890625! disk.dm-0.out‚118458.29296875 disk.sda.out‚1287.587890625$ memory_percent‚91.96619987487793* proc.memory.percent‚18.214964866638184 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12109375! disk.nvme0n1.out‚ 92480.890625 disk.md0.out‚ 1257.5390625# proc.memory.rssMB‚ 2984.33984375 cpu‚25.231514961430566) disk./.usagePercent‚75.68693009144476 disk.rbd0.out‚ 2.25439453125 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.sdb.out‚1287.587890625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1257.5390625 disk.sda1.in‚0 disk.sda.in‚128.625 proc.cpu.threads‚17 network.sent‚ 1.6092235e+07 network.recv‚ 2.545491e+06 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 19.140625# disk.nvme1n1.out‚25991.02734375 disk.rbd0.in‚0)è²’æ‚(:‡ £«ëÔë–Ùà gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<7êgç‚(:Ü £«ëÔɬØá) disk./.usagePercent‚75.68700507379886 network.recv‚ 2.550689e+06 disk.rbd0.out‚ 2.25439453125 disk.rbd2.out‚ 2.466796875 disk.sda2.out‚ 1288.99609375( proc.memory.availableMB‚ 1316.2578125 proc.cpu.threads‚17 disk.sda.out‚ 1288.99609375 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.sda1.in‚0 disk.dm-0.in‚ 24.26171875 cpu‚25.231088674505735 disk.sda.in‚128.625 disk.nvme1n1.in‚ 19.140625 disk.md0p1.out‚ 1258.90625 disk.rbd0.in‚0* proc.memory.percent‚18.214964866638184 disk.md0.in‚ 199.5859375$ disk./.usageGB‚2706.933910369873 disk.sdb2.out‚ 1288.99609375 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 network.sent‚ 1.6186218e+07# disk.nvme1n1.out‚25991.22265625! disk.nvme0n1.out‚ 92653.109375 disk.md0.out‚ 1258.90625! disk.dm-0.out‚118644.33203125 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 1288.99609375$ memory_percent‚91.96619987487793 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12109375# proc.memory.rssMB‚ 2984.33984375±¢Ì•”è‚(:‰ ²«ëÔ« Êà gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐSS§wé‚(:ì ²«ëÔ÷ˆá disk.sda.in‚128.625 disk.md0p1.out‚ 1260.3828125) disk./.usagePercent‚75.69400712441158" disk.sda2.out‚1290.52490234375 disk.md0.in‚ 199.5859375! disk.sda.out‚1290.52490234375 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 5.12109375# proc.memory.rssMB‚ 2984.33984375 network.recv‚ 2.554839e+06 disk.nvme1n1.in‚ 19.140625 disk.rbd2.in‚0 disk.rbd0.out‚ 2.31689453125 disk.sdb1.in‚0 network.sent‚ 1.6191927e+07 disk.sda2.in‚128.625! disk.sdb.out‚1290.52490234375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.md0.out‚ 1260.3828125" disk.sdb2.out‚1290.52490234375 disk.sda1.out‚0" disk.nvme0n1.out‚ 93015.9453125$ memory_percent‚91.96622371673584) proc.memory.availableMB‚ 1316.25390625 cpu‚25.2306672327198 proc.cpu.threads‚17 disk.dm-0.in‚ 24.26171875! disk.dm-0.out‚119007.48046875 disk.sda1.in‚0* proc.memory.percent‚18.214964866638184$ disk./.usageGB‚2707.184337615967 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚25991.53515625 disk.sdb1.out‚0¿q¨1”ê‚(:‰ Á«ëÔòœÃà gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü%!å{ë‚(:ð Á«ëÔ¯—™á network.sent‚ 1.6200274e+07" disk.sda2.out‚1291.97900390625 disk.dm-0.in‚ 24.26171875 disk.sda.in‚128.625 disk.nvme1n1.in‚ 19.140625) proc.memory.availableMB‚ 1316.25390625* proc.memory.percent‚18.214964866638184 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 5.12109375# disk.nvme0n1.out‚93120.07421875 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625! disk.dm-0.out‚119111.80859375! disk.sda.out‚1291.97900390625! disk.nvme1n1.out‚ 25991.734375# proc.memory.rssMB‚ 2984.33984375) disk./.usagePercent‚75.69398931210272 disk.rbd0.out‚ 2.31689453125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 network.recv‚ 2.559602e+06% disk./.usageGB‚2707.1837005615234 cpu‚25.230275297277718 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1261.76171875 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1291.97900390625 disk.md0p1.in‚ 199.5859375 disk.md0p1.out‚ 1261.76171875 disk.md0.in‚ 199.5859375! disk.sdb.out‚1291.97900390625$ memory_percent‚91.96622371673584uäzi2ì- _wandbruntime‚7547  _runtime‚7547t#³Åí¿ & lr-AdamW/pg1‚0.0004028646487507811 & lr-AdamW/pg2‚0.0004028646487507811 trainer/global_step‚6299 $ _timestamp‚1.788532164252296e+09 _runtime‚ 7547.78864542 _step‚515?¢+éÊîÄ trainer/global_step‚6299 $ _timestamp‚1.788532164252296e+09 _runtime‚ 7547.78864542 _step‚515 & lr-AdamW/pg1‚0.0004028646487507811 & lr-AdamW/pg2‚0.0004028646487507811ƒ“ã#2ï- _wandbruntime‚7547  _runtime‚7547tÿÏôû'ðõO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚ 7547.79083535 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7221946605568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538537301832192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131893734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531316283201024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513716903375984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7221036634112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚593712925239808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586167207904768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7545693394944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159633236 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593712687197696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79028580 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131893734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531316283201024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520911867388100 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131893732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885321642544844e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚538538229806592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586167231844864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7194946972756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7545479292928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7221036634112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38305751 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚538538229806592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538537301832192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7221946605568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538537301832192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131893734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531316283201024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513716903375984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7221036634112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚593712925239808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586167207904768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7545693394944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159633236 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593712687197696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79028580 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131893734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531316283201024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520911867388100 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131893732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885321642544844e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚538538229806592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586167231844864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7194946972756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7545479292928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7221036634112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38305751 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚538538229806592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538537301832192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7221946605568 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚593804319451136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7195156271030 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586257457549312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159658067 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131913894 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117352690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291598331 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531397884283904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131913892 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159658067 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117352068 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885321653990595e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531397884283904 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531397902286848 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159684437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531397902286848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159684437 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79040643 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291571959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚7548.935519579 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7196064246242 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6299ðÐB‡ 'ô…N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚7548.935519579 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7196064246242 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291598331 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7223068079104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38311425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7546861901824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513795920970528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520991077241558 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚513795938009888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚593804319451136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7195156271030 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586257457549312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159658067 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131913894 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117352690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291598331 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531397884283904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131913892 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159658067 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117352068 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885321653990595e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531397884283904 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531397902286848 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159684437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531397902286848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159684437 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79040643 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291571959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144…µZ32õ- _wandbruntime‚7548  _runtime‚7548]Ã"^öØ /loss/FRt3/detection_loss‚1.3828633667726535 1loss/SRqqt3/detection_loss‚0.8579084169557866 epoch‚5 _runtime‚7548.936470803 &loss/total_loss‚42.055557106293605 0loss/FRt3/assignment_loss‚1.0394901130348444 0loss/FRt4/assignment_loss‚3.1962853060828316 2loss/SRqqt2/assignment_loss‚1.3803518499646867 0loss/SRqqt4/detection_loss‚2.262856803157113 /loss/FBt1/detection_loss‚2.8512369069186123 /loss/FBt3/detection_loss‚0.6589518537049193 .loss/FBt4/detection_loss‚2.590766886870066 2loss/SRqqt1/assignment_loss‚1.8280777561087762 1loss/FBt1/assignment_loss‚0.35187848925935133 /loss/FBt2/assignment_loss‚0.307059025932436 /loss/FRt1/detection_loss‚2.6089947098179866 1loss/SRqqt1/detection_loss‚2.3748811867929276 2loss/SRqqt4/assignment_loss‚1.4137781014045079 1loss/SRqqt2/detection_loss‚2.1529354442443167 /loss/FBt2/detection_loss‚2.5828512736729214 % _timestamp‚1.7885321654011717e+09 /loss/FRt1/assignment_loss‚3.435663747160058 0loss/FRt2/assignment_loss‚2.9456887041650166 2loss/SRqqt3/assignment_loss‚0.9345148932407884 /loss/FRt4/detection_loss‚2.2391012077530226 _step‚518 .loss/FRt2/detection_loss‚2.482736657608745 trainer/global_step‚6299 1loss/FBt3/assignment_loss‚0.08465676891409067 1loss/FBt4/assignment_loss‚0.09202763675714071ªúú’c÷Ý /loss/FBt3/detection_loss‚0.6589518537049193 .loss/FBt4/detection_loss‚2.590766886870066 2loss/SRqqt1/assignment_loss‚1.8280777561087762 1loss/FBt1/assignment_loss‚0.35187848925935133 /loss/FBt2/assignment_loss‚0.307059025932436 /loss/FRt1/detection_loss‚2.6089947098179866 1loss/SRqqt1/detection_loss‚2.3748811867929276 2loss/SRqqt4/assignment_loss‚1.4137781014045079 1loss/SRqqt2/detection_loss‚2.1529354442443167 /loss/FBt2/detection_loss‚2.5828512736729214 % _timestamp‚1.7885321654011717e+09 /loss/FRt1/assignment_loss‚3.435663747160058 0loss/FRt2/assignment_loss‚2.9456887041650166 2loss/SRqqt3/assignment_loss‚0.9345148932407884 /loss/FRt4/detection_loss‚2.2391012077530226 _step‚518 .loss/FRt2/detection_loss‚2.482736657608745 trainer/global_step‚6299 1loss/FBt3/assignment_loss‚0.08465676891409067 1loss/FBt4/assignment_loss‚0.09202763675714071 /loss/FRt3/detection_loss‚1.3828633667726535 1loss/SRqqt3/detection_loss‚0.8579084169557866 epoch‚5 _runtime‚7548.936470803 &loss/total_loss‚42.055557106293605 0loss/FRt3/assignment_loss‚1.0394901130348444 0loss/FRt4/assignment_loss‚3.1962853060828316 2loss/SRqqt2/assignment_loss‚1.3803518499646867 0loss/SRqqt4/detection_loss‚2.262856803157113 /loss/FBt1/detection_loss‚2.8512369069186123†ª—µ“ø‚(:ˆ ЫëÔó¬§á gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h±ÑŠlù‚(:á ЫëÔ¸çá disk.sda.in‚128.625 disk.sda.out‚1293.865234375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚128.625$ memory_percent‚91.96197986602783) proc.memory.availableMB‚ 1316.94921875% disk./.usageGB‚2707.4346771240234 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1293.865234375 disk.nvme1n1.in‚ 19.140625 disk.rbd2.out‚ 2.466796875 disk.md0p1.in‚ 199.5859375 disk.md0.in‚ 199.5859375 disk.dm-0.out‚119328.3828125 disk.md0p1.out‚ 1263.59375 disk.sda2.out‚1293.865234375# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824# disk.nvme0n1.out‚93336.10546875 network.sent‚ 1.6217204e+07 disk.md0.out‚ 1263.59375 disk.sdb.out‚1293.865234375 disk.sda1.in‚0 disk.nvme0n1.in‚ 5.12109375 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.22988151210173 proc.cpu.threads‚17 disk.dm-0.in‚ 24.26171875 disk.rbd0.out‚ 2.3408203125 network.recv‚ 2.564234e+06) disk./.usagePercent‚75.70100672183206 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚25992.27734375 disk.sdb1.in‚0sY-”ú‚(:‰ ß«ëÔíïÓà gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…YIkrû‚(:ç ß«ëÔöÓ™á network.recv‚ 2.569372e+06 disk.sdb1.out‚0 disk.sdb.out‚1295.5673828125! disk.sdb2.out‚1295.5673828125( proc.memory.availableMB‚ 1317.1953125 cpu‚25.229497797243212$ disk./.usageGB‚2707.434585571289# disk.nvme0n1.out‚93760.83203125 disk.dm-0.out‚ 119753.28125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12109375) disk./.usagePercent‚75.70100416197928 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1265.22265625 disk.sdb1.in‚0 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.sda.out‚1295.5673828125* proc.memory.percent‚18.208956718444824! disk.sda2.out‚1295.5673828125# disk.nvme1n1.out‚25992.44921875$ memory_percent‚91.96047782897949 disk.rbd2.out‚ 2.466796875 disk.nvme1n1.in‚ 19.140625# proc.memory.rssMB‚ 2983.35546875 proc.cpu.threads‚17 network.sent‚ 1.631133e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 2.3408203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1265.22265625 disk.dm-0.in‚ 24.26171875 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375aHËX“ü‚(:ˆ î«ëÔÎóˆà gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­P2xý‚(:í î«ëÔà˜á! disk.dm-0.out‚119925.69921875) disk./.usagePercent‚75.70100416197928 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375) proc.memory.availableMB‚ 1316.94921875 cpu‚25.229078992599582 disk.rbd0.out‚ 2.40380859375 disk.rbd2.out‚ 2.466796875! disk.sdb.out‚1296.96533203125! disk.sda.out‚1296.96533203125 disk.md0p1.out‚ 1266.5703125 network.recv‚ 2.573203e+06 network.sent‚ 1.6317094e+07 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 24.26171875" disk.sdb2.out‚1296.96533203125# proc.memory.rssMB‚ 2983.35546875! disk.nvme0n1.out‚ 93933.046875 disk.rbd0.in‚0 disk.sdb1.out‚0" disk.sda2.out‚1296.96533203125 disk.sda.in‚128.625# disk.nvme1n1.out‚25992.65234375 disk.md0p1.in‚ 199.5859375$ memory_percent‚91.96197986602783 disk.md0.out‚ 1266.5703125 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.208956718444824 disk.rbd2.in‚0$ disk./.usageGB‚2707.434585571289 disk.sdb1.in‚0 disk.sda2.in‚128.625 disk.nvme1n1.in‚ 19.140625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 5.12109375α‚דþ‚(:ˆ ý«ëÔêмà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ÝÈŽmÿ‚(:â ý«ëÔàÕ¢á disk.sda2.in‚128.625 disk.sda.in‚128.625 disk.sda.out‚1298.806640625 cpu‚25.22866030524943 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875 disk.md0.in‚ 199.5859375 disk.sdb1.out‚0# proc.memory.rssMB‚ 2983.35546875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12890625 disk.md0.out‚ 1268.3515625 disk.md0p1.in‚ 199.5859375 disk.sda1.in‚0( proc.memory.availableMB‚ 1317.1953125 disk.sdb1.in‚0 disk.dm-0.out‚ 120126.09375 network.sent‚ 1.6323287e+07 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1268.3515625 disk.rbd0.in‚0# disk.nvme0n1.out‚94133.29296875 disk.sda2.out‚1298.806640625* proc.memory.percent‚18.208956718444824# disk.nvme1n1.out‚25992.80078125 disk.rbd0.out‚ 2.40380859375 network.recv‚ 2.577858e+06$ disk./.usageGB‚2706.904739379883 disk.dm-0.in‚ 24.26953125 disk.sdb.out‚1298.806640625 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1298.806640625 disk.nvme1n1.in‚ 19.140625$ memory_percent‚91.96047782897949) disk./.usagePercent‚75.68618944070971 µR2€- _wandbruntime‚7605  _runtime‚7605ñÕŠ Éà ' lr-AdamW/pg1‚0.00039760582401666495 ' lr-AdamW/pg2‚0.00039760582401666495 trainer/global_step‚6349 % _timestamp‚1.7885322221670947e+09 _runtime‚7605.702734158 _step‚519挋jÎ‚È ' lr-AdamW/pg2‚0.00039760582401666495 trainer/global_step‚6349 % _timestamp‚1.7885322221670947e+09 _runtime‚7605.702734158 _step‚519 ' lr-AdamW/pg1‚0.00039760582401666495‡~},2ƒ- _wandbruntime‚7605  _runtime‚7605VN…ð „öO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524917699356044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118226348 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚535397219489280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293806216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38594520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542675271522816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542674343548416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118226988 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590678517067264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524918624370408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7277124059136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598282408964096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7249916252860 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚7605.704431435 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132901732 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160904482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7603915836928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590678493127168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7250824227864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160878138 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7277124059136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598282661686272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚6349 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535397237492224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚535397219489280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132901732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160878138 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517667783103184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160904482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542674343548416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7277124059136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598282408964096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7249916252860 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚7605.704431435 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132901732 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160904482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7603915836928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590678493127168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7250824227864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160878138 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7277124059136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598282661686272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚6349 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535397237492224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚535397219489280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132901732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160878138 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517667783103184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160904482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542674343548416 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293851233 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542757084094464 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160902969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160902969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7605069285888 % _timestamp‚1.7885322233299854e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278245519360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293851233 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160929339 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279155504128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118244125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590768742771712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590768718831616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7251033526138 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535478838575104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643895 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524997834223866 _runtime‚7606.866316028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚517746800697728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598373788117504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132921892 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535478838575104Sï¢ 'ˆ…N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132921892 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535478838575104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535478856578048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚535478856578048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 trainer/global_step‚6349 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38600232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598374051312128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293824861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38600879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7605308540416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524998759238438 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132921892 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293851233 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542757084094464 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160902969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160902969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7605069285888 % _timestamp‚1.7885322233299854e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278245519360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293851233 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160929339 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279155504128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118244125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590768742771712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590768718831616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7251033526138 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535478838575104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643895 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524997834223866 _runtime‚7606.866316028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚517746800697728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598373788117504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570‰wTªJ2‰- _wandbruntime‚7606  _runtime‚76063yw+MŠÓ /loss/FRt1/assignment_loss‚2.896076648700528 1loss/SRqqt4/detection_loss‚2.1413047321744867 epoch‚5 _step‚522 /loss/FRt2/assignment_loss‚3.348355633241159 .loss/FRt1/detection_loss‚2.564321352214348 /loss/FBt3/detection_loss‚0.5951146811971234 % _timestamp‚1.7885322233319793e+09 0loss/FRt4/assignment_loss‚3.1003866034585075 2loss/SRqqt1/assignment_loss‚1.3218890137293122 1loss/FBt1/assignment_loss‚0.09136991630013934 0loss/FBt4/assignment_loss‚0.1810643169209925 .loss/FBt1/detection_loss‚2.404325778667743 .loss/FBt2/detec.h¾ó tion_loss‚2.217163842916489 .loss/FBt4/detection_loss‚2.208721876144409 $loss/total_loss‚39.6898933635828 1loss/SRqqt4/assignment_loss‚1.537175241455033 2loss/FBt3/assignment_loss‚0.019404878464659388 /loss/FRt2/detection_loss‚2.5008624266491903 /loss/FRt4/detection_loss‚2.5703003117059535 1loss/SRqqt3/detection_loss‚0.9118830216127437 trainer/global_step‚6349 _runtime‚7606.870948147 2loss/SRqqt3/assignment_loss‚0.8835611505495323 1loss/FBt2/assignment_loss‚0.12313772746711038 /loss/FRt3/detection_loss‚1.3392149725038072 /loss/SRqqt2/detection_loss‚2.60787904049669 0loss/FRt3/assignment_loss‚1.0040753172791523 2loss/SRqqt2/assignment_loss‚1.1464603881218605 1loss/SRqqt1/detection_loss‚1.9758444916118274`F‰^‹Ø 0loss/FRt3/assignment_loss‚1.0040753172791523 2loss/SRqqt2/assignment_loss‚1.1464603881218605 1loss/SRqqt1/detection_loss‚1.9758444916118274 /loss/FRt1/assignment_loss‚2.896076648700528 1loss/SRqqt4/detection_loss‚2.1413047321744867 epoch‚5 _step‚522 /loss/FRt2/assignment_loss‚3.348355633241159 .loss/FRt1/detection_loss‚2.564321352214348 /loss/FBt3/detection_loss‚0.5951146811971234 % _timestamp‚1.7885322233319793e+09 0loss/FRt4/assignment_loss‚3.1003866034585075 2loss/SRqqt1/assignment_loss‚1.3218890137293122 1loss/FBt1/assignment_loss‚0.09136991630013934 0loss/FBt4/assignment_loss‚0.1810643169209925 .loss/FBt1/detection_loss‚2.404325778667743 .loss/FBt2/detection_loss‚2.217163842916489 .loss/FBt4/detection_loss‚2.208721876144409 $loss/total_loss‚39.6898933635828 1loss/SRqqt4/assignment_loss‚1.537175241455033 2loss/FBt3/assignment_loss‚0.019404878464659388 /loss/FRt2/detection_loss‚2.5008624266491903 /loss/FRt4/detection_loss‚2.5703003117059535 1loss/SRqqt3/detection_loss‚0.9118830216127437 trainer/global_step‚6349 _runtime‚7606.870948147 2loss/SRqqt3/assignment_loss‚0.8835611505495323 1loss/FBt2/assignment_loss‚0.12313772746711038 /loss/FRt3/detection_loss‚1.3392149725038072 /loss/SRqqt2/detection_loss‚2.60787904049669Š|L/Ÿ“Œ‚(:ˆ Œ¬ëÔ£ù”à gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05{nºz‚(:ï Œ¬ëÔáîßá disk.md0.in‚ 199.5859375" disk.sdb2.out‚1300.30712890625) proc.memory.availableMB‚ 1317.16796875 disk.sdb1.in‚0 disk.sda.in‚128.625 cpu‚25.228240979064026 network.recv‚ 2.582245e+06 disk.sda1.out‚0 disk.md0.out‚ 1269.79296875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚18.208956718444824 disk.md0p1.out‚ 1269.79296875 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.dm-0.out‚120443.3359375! disk.sdb.out‚1300.30712890625$ memory_percent‚91.96064472198486 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0# disk.nvme0n1.out‚94450.21484375 disk.dm-0.in‚ 24.26953125 proc.cpu.threads‚17% disk./.usageGB‚2706.9044914245605 disk.nvme1n1.in‚ 19.140625) disk./.usagePercent‚75.68618250777513# disk.nvme1n1.out‚25993.12109375 disk.sda2.in‚128.625" disk.sda2.out‚1300.30712890625! disk.sda.out‚1300.30712890625# proc.memory.rssMB‚ 2983.35546875 network.sent‚ 1.6342351e+07 disk.sda1.in‚0 disk.nvme0n1.in‚ 5.12890625FŠ”Ž‚(:‰ ›¬ëÔôîà gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈÈŽÂq‚(:æ ›¬ëÔ´¹¡á disk.dm-0.in‚ 24.26953125 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 2.466796875& proc.memory.availableMB‚ 1317.59375 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875! disk.nvme0n1.out‚ 94586.921875! disk.dm-0.out‚120580.26171875 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1301.65771484375# disk.nvme1n1.out‚25993.33984375" disk.sdb2.out‚1301.65771484375# proc.memory.rssMB‚ 2983.35546875 network.recv‚ 2.587283e+06$ disk./.usageGB‚2706.847816467285 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.rbd0.in‚0* proc.memory.percent‚18.208956718444824 disk.sda1.in‚0 disk.rbd2.in‚0 cpu‚25.22786265794836( disk./.usagePercent‚75.6845978522494 disk.nvme1n1.in‚ 19.140625 disk.sda2.in‚128.625" disk.sda2.out‚1301.65771484375 disk.sda.in‚128.625 disk.md0p1.out‚ 1271.1015625 network.sent‚ 1.6436195e+07 disk.sdb1.out‚0 disk.md0.out‚ 1271.1015625 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1301.65771484375$ memory_percent‚91.95804595947266 disk.sdb1.in‚0 disk.nvme0n1.in‚ 5.12890625c b¡‘‚(:† ª¬ëÔÆ«²à gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÞÿ@r‘‚(:ç ª¬ëÔ»ú•á disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0& proc.memory.availableMB‚ 1317.34375 disk.rbd2.in‚0" disk.sdb2.out‚1303.34423828125 disk.sda2.in‚128.625! disk.sda.out‚1303.34423828125 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 disk.dm-0.in‚ 24.26953125 disk.sdb1.out‚0 disk.sdb1.in‚0" disk.sda2.out‚1303.34423828125 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.nvme0n1.out‚ 94746.90625# memory_percent‚91.9595718383789 network.recv‚ 2.590865e+06 disk.md0.out‚ 1272.72265625! disk.sdb.out‚1303.34423828125 disk.sda.in‚128.625" disk.nvme1n1.out‚ 25993.5859375 disk.sda1.in‚0% disk./.usageGB‚2706.9346809387207 disk.nvme1n1.in‚ 19.140625 disk.md0.in‚ 199.5859375 disk.md0p1.out‚ 1272.72265625 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 5.12890625 cpu‚25.227450447011297 proc.cpu.threads‚17 network.sent‚ 1.6444057e+07) disk./.usagePercent‚75.68702661922634 disk.dm-0.out‚120740.4921875`ŒÐ2’- _wandbruntime‚7662  _runtime‚76626`ŽÂÇ“Á _step‚523 & lr-AdamW/pg1‚0.0003923403917816338 & lr-AdamW/pg2‚0.0003923403917816338 trainer/global_step‚6399 % _timestamp‚1.7885322794638865e+09 _runtime‚7662.999434183s†ŸÌ”Æ & lr-AdamW/pg1‚0.0003923403917816338 & lr-AdamW/pg2‚0.0003923403917816338 trainer/global_step‚6399 % _timestamp‚1.7885322794638865e+09 _runtime‚7662.999434183 _step‚523‹KÕ†Ø2•- _wandbruntime‚7663  _runtime‚7663c¿ü'–öO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546811385264640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133909732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602851835734528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7305793507968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885322794646184e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133909734 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546812313239040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119118565 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7662057384960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296059118 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162123040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚521618679869744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7334121455616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7306701482972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296032772 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚539478191783424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602852090553856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521618662830384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚539478191783424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚539478173780480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚595189778349568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546812313239040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162149384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162149384 trainer/global_step‚6399 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133909732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80235080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162123040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133909734 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333211484160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚595189802289664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546811385264640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38882846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7334121455616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7662288264192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚7663.000273117 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚539478173780480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784ŸÎ(—ûO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133909734 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546812313239040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119118565 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7662057384960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296059118 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162123040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚521618679869744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7334121455616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7306701482972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296032772 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚539478191783424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602852090553856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521618662830384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚539478191783424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚539478173780480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚595189778349568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546812313239040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162149384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162149384 trainer/global_step‚6399 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133909732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80235080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162123040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133909734 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333211484160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚595189802289664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546811385264640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38882846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7334121455616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7662288264192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚7663.000273117 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚539478173780480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546811385264640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133909732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602851835734528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7305793507968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885322794646184e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192ŒùêÙ2˜- _wandbruntime‚7664  _runtime‚7664Gk<ë™þM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162174241 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚539559792866304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119136267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296104135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595280004054016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334332944384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XDDeviceSF ‰tatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602943215510528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133929894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38889122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80247145 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529004591206174 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133929892 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162174241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚521697697464288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚521697680424928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296104135 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602943478705152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚539559810869248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335242929152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38888488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546895053798400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119135631 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7664.028831284 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7663450711040±s‹‹ 'šƒN JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546895053798400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119135631 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7664.028831284 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7663450711040 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162174241 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚539559792866304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119136267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296104135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595280004054016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334332944384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602943215510528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133929894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38889122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80247145 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529004591206174 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133929892 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162174241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚521697697464288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚521697680424928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296104135 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602943478705152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚539559810869248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335242929152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38888488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521es“2›- _wandbruntime‚7664  _runtime‚76645ƽ\œÖ 2loss/SRqqt1/assignment_loss‚1.5802408980860099 2loss/SRqqt4/assignment_loss‚1.4116445968345719 2loss/FBt3/assignment_loss‚0.046830217320088095 /loss/FBt3/detection_loss‚0.6701055190500778 trainer/global_step‚6399 % _timestamp‚1.7885322804948533e+09 _runtime‚7664.030305301 0loss/FRt3/assignment_loss‚1.1045664093560643 /loss/FBt1/assignment_loss‚0.318920989907513 /loss/FRt1/detection_loss‚2.9733505495663346 /loss/FRt2/detection_loss‚2.6441968347384304 1loss/SRqqt3/detection_loss‚0.8530386286703023 1loss/SRqqt2/detection_loss‚2.4902769466862082 /loss/FBt1/detection_loss‚2.6614900401660373 _step‚526 1loss/FBt2/assignment_loss‚0.23546073681207816 /loss/FRt4/detection_loss‚2.1700949191273944 1loss/SRqqt4/detection_loss‚1.9137764962352053 $loss/total_loss‚41.9488301842407 epoch‚5 0loss/FRt1/assignment_loss‚3.6463772264020196 0loss/FRt2/assignment_loss‚3.6576746590435505 .loss/FRt3/detection_loss‚1.404278661434849 .loss/FBt2/detection_loss‚2.265655982494354 /loss/FRt4/assignment_loss‚3.125295989918259 2loss/SRqqt3/assignment_loss‚1.0244097566100383 0loss/SRqqt1/detection_loss‚2.379155914525728 /loss/FBt4/detection_loss‚1.8569712489843369 2loss/SRqqt2/assignment_loss‚1.3300999391358346 0loss/FBt4/assignment_loss‚0.1849170231354158œ:nMaÛ 2loss/SRqqt3/assignment_loss‚1.0244097566100383 0loss/SRqqt1/detection_loss‚2.379155914525728 /loss/FBt4/detection_loss‚1.8569712489843369 2loss/SRqqt2/assignment_loss‚1.3300999391358346 0loss/FBt4/assignment_loss‚0.1849170231354158 2loss/SRqqt1/assignment_loss‚1.5802408980860099 2loss/SRqqt4/assignment_loss‚1.4116445968345719 2loss/FBt3/assignment_loss‚0.046830217320088095 /loss/FBt3/detection_loss‚0.6701055190500778 trainer/global_step‚6399 % _timestamp‚1.7885322804948533e+09 _runtime‚7664.030305301 0loss/FRt3/assignment_loss‚1.1045664093560643 /loss/FBt1/assignment_loss‚0.318920989907513 /loss/FRt1/detection_loss‚2.9733505495663346 /loss/FRt2/detection_loss‚2.6441968347384304 1loss/SRqqt3/detection_loss‚0.8530386286703023 1loss/SRqqt2/detection_loss‚2.4902769466862082 /loss/FBt1/detection_loss‚2.6614900401660373 _step‚526 1loss/FBt2/assignment_loss‚0.23546073681207816 /loss/FRt4/detection_loss‚2.1700949191273944 1loss/SRqqt4/detection_loss‚1.9137764962352053 $loss/total_loss‚41.9488301842407 epoch‚5 0loss/FRt1/assignment_loss‚3.6463772264020196 0loss/FRt2/assignment_loss‚3.6576746590435505 .loss/FRt3/detection_loss‚1.404278661434849 .loss/FBt2/detection_loss‚2.265655982494354 /loss/FRt4/assignment_loss‚3.125295989918259Žh´¶’ž‚(:‡ ¹¬ëÔè¼à gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰õó%hŸ‚(:Ý ¹¬ëÔª‰¤á) disk./.usagePercent‚75.69052668458791 disk.dm-0.in‚ 24.26953125 disk.dm-0.out‚ 120985.546875 disk.md0p1.out‚ 1274.49609375 disk.sdb1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚94991.81640625 disk.sda2.in‚128.625 network.recv‚ 2.595544e+06$ disk./.usageGB‚2707.059860229492# disk.nvme1n1.out‚25993.73046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 5.12890625 disk.md0.out‚ 1274.49609375 disk.nvme1n1.in‚ 19.140625 disk.rbd0.out‚ 2.466796875# memory_percent‚91.9611930847168' proc.memory.availableMB‚ 1317.078125 disk.sdb.out‚ 1305.18359375 disk.md0p1.in‚ 199.5859375 disk.sda2.out‚ 1305.18359375# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 cpu‚25.22710352081355 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1305.18359375 disk.sda.in‚128.625 disk.sda.out‚ 1305.18359375 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 network.sent‚ 1.6448162e+07 disk.md0.in‚ 199.5859375—ˆ§ž“ ‚(:ˆ ȬëÔµù¹à gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ<8õr¡‚(:ç ȬëÔª‘˜á disk.sda.in‚128.625 disk.rbd2.in‚0* proc.memory.percent‚18.208956718444824% disk./.usageGB‚2707.1845932006836 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1306.5458984375 disk.md0p1.out‚ 1275.80859375 disk.nvme1n1.in‚ 19.140625! disk.sda2.out‚1306.5458984375 network.recv‚ 2.601237e+06! disk.sdb2.out‚1306.5458984375" disk.nvme1n1.out‚ 25994.0234375 disk.rbd2.out‚ 2.466796875 proc.cpu.threads‚17! disk.nvme0n1.out‚ 95299.328125 disk.dm-0.in‚ 24.26953125 disk.dm-0.out‚121293.3515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 5.12890625 disk.rbd0.in‚0 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1275.80859375$ memory_percent‚91.95802211761475# proc.memory.rssMB‚ 2983.35546875) disk./.usagePercent‚75.69401427066722 disk.sdb.out‚1306.5458984375 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚128.625) proc.memory.availableMB‚ 1317.59765625 cpu‚25.22672687017271 network.sent‚ 1.6558007e+07atТ‚(:’ ׬ëÔËû”à gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýœ yu£‚(:ê ׬ëÔý½Ëà network.sent‚ 1.6563377e+07% disk./.usageGB‚2707.5596199035645 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12890625! disk.sdb.out‚1308.08544921875 disk.md0p1.out‚ 1277.2890625# proc.memory.rssMB‚ 2983.35546875 cpu‚25.226322732891017 network.recv‚ 2.60506e+06 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1277.2890625! disk.sda.out‚1308.08544921875) disk./.usagePercent‚75.70450017424064 disk.sda1.in‚0 disk.dm-0.in‚ 24.26953125 disk.nvme1n1.in‚ 19.140625 proc.cpu.threads‚17! disk.dm-0.out‚121697.34765625( proc.memory.availableMB‚ 1317.6015625 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚18.208956718444824 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1308.08544921875 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0! disk.nvme0n1.out‚ 95586.703125 disk.rbd0.out‚ 2.466796875$ memory_percent‚91.95799827575684 disk.sda2.in‚128.625" disk.sda2.out‚1308.08544921875 disk.md0.in‚ 199.5859375# disk.nvme1n1.out‚26110.64453125 disk.rbd0.in‚0a4"䔤‚(:‰ æ¬ëÔ¯—à gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šgTb¥‚(:× æ¬ëÔü¬à disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0$ memory_percent‚91.95802211761475# proc.memory.rssMB‚ 2983.35546875 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 network.recv‚ 2.608897e+06 disk.dm-0.in‚ 24.26953125 disk.nvme1n1.in‚ 19.140625# disk.nvme1n1.out‚26110.85546875 network.sent‚ 1.6568998e+07 disk.sda2.out‚ 1309.953125 disk.md0.out‚ 1279.109375% disk./.usageGB‚2707.4345932006836 disk.rbd2.in‚0 disk.sda.out‚ 1309.953125* proc.memory.percent‚18.208956718444824 proc.cpu.threads‚17 disk.sda2.in‚128.625! disk.dm-0.out‚121846.78515625 disk.sdb.out‚ 1309.953125 disk.sdb2.out‚ 1309.953125 disk.md0p1.out‚ 1279.109375 disk.sdb1.out‚0 cpu‚25.225923096684195) disk./.usagePercent‚75.70100437530036 disk.nvme0n1.in‚ 5.12890625" disk.nvme0n1.out‚ 95735.9296875) proc.memory.availableMB‚ 1317.59765625Áq¶2¦- _wandbruntime‚7720  _runtime‚77202‚NǧÁ _step‚527 & lr-AdamW/pg1‚0.0003870694493538735 & lr-AdamW/pg2‚0.0003870694493538735 trainer/global_step‚6449 % _timestamp‚1.7885323370712209e+09 _runtime‚7720.606749958µÇlÌ¨Æ _runtime‚7720.606749958 _step‚527 & lr-AdamW/pg1‚0.0003870694493538735 & lr-AdamW/pg2‚0.0003870694493538735 trainer/global_step‚6449 % _timestamp‚1.7885323370712209e+09$¿ü 2©- _wandbruntime‚7720  _runtime‚7720ï a€sªöO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7390208880640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80838330 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚525569559596944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7362578738080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298285674 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134917734 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120010608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550949354955264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134917734 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532932138335024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134917732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163367942 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚6449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550948426980864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543559146074624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532931213320660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550948426980864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80838328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163394286 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on©ŽH‰_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7361670763076 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134917732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599701063571968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚525569542557584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543559146074624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550949354955264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7720560273920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163394286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7389298909184–Q7(«ûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298285674 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134917734 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120010608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550949354955264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134917734 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532932138335024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134917732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163367942 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚6449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550948426980864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543559146074624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532931213320660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550948426980864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80838328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163394286 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7361670763076 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134917732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599701063571968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚525569542557584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543559146074624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550949354955264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7720560273920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163394286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7389298909184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7390208880640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80838330 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚525569559596944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7362578738080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570h-7À2¬- _wandbruntime‚7721  _runtime‚7721°ÊÓ'­ÿM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7721955610112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚6449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551032095514624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7721726841344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551031167526912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163392773 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607513016117760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39178029 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7390420369408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134937894 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7390420369408 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120027810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39177417 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599791289276416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80850393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7391330354176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525648577191488 _step‚529 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134937892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533011348188482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525648560152128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120028424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885323382268844e+09 _runtime‚ 7721.76307213 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298357037 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298330665 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚533012273203054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551031167526912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163392773 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607513016117760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39178029 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7390420369408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134937894 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7390420369408 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120027810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39177417 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599791289276416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80850393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7391330354176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525648577191488 _step‚529 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134937892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533011348188482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525648560152128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120028424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885323382268844e+09 _runtime‚ 7721.76307213 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298357037 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298330665 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚533012273203054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144‘Z®Š2¯- _wandbruntime‚7721  _runtime‚7721ì_ýÒ°Ø 2loss/SRqqt1/assignment_loss‚1.2644234609145384 1loss/SRqqt1/detection_loss‚2.4793337491842418 trainer/global_step‚6449 0loss/FRt1/assignment_loss‚2.4160116856748406 2loss/SRqqt2/assigÏqºŒnment_loss‚1.8107786443855227 2loss/SRqqt3/assignment_loss‚1.0991591440830777 2loss/SRqqt4/assignment_loss‚1.2368341173308017 1loss/FBt2/assignment_loss‚0.11306801477717972 /loss/FRt4/detection_loss‚2.2144837448034775 /loss/FRt2/assignment_loss‚3.016843926262211 1loss/FBt4/assignment_loss‚0.09416787175314312 0loss/SRqqt2/detection_loss‚2.549095033571638 1loss/FBt1/assignment_loss‚0.15950549288221996 0loss/FBt3/assignment_loss‚0.1180146139479778 .loss/FRt2/detection_loss‚2.384869142026071 &loss/total_loss‚41.741406440755945 _runtime‚7721.764668927 /loss/FBt2/detection_loss‚2.5364214594547567 /loss/FBt3/detection_loss‚0.5490173251695327 _step‚530 epoch‚5 1loss/SRqqt3/detection_loss‚0.8767239922617979 % _timestamp‚1.7885323382290947e+09 0loss/FRt3/assignment_loss‚1.2895768965993608 0loss/FRt4/assignment_loss‚3.5098980854858053 .loss/FRt1/detection_loss‚2.765980120861169 /loss/FRt3/detection_loss‚1.4576566404411717 1loss/SRqqt4/detection_loss‚2.1254074420684423 /loss/FBt1/detection_loss‚2.4709478744438718 .loss/FBt4/detection_loss‚3.203187962373098ŽPe£c±Ý 2loss/SRqqt1/assignment_loss‚1.2644234609145384 1loss/SRqqt1/detection_loss‚2.4793337491842418 trainer/global_step‚6449 0loss/FRt1/assignment_loss‚2.4160116856748406 2loss/SRqqt2/assignment_loss‚1.8107786443855227 2loss/SRqqt3/assignment_loss‚1.0991591440830777 2loss/SRqqt4/assignment_loss‚1.2368341173308017 1loss/FBt2/assignment_loss‚0.11306801477717972 /loss/FRt4/detection_loss‚2.2144837448034775 /loss/FRt2/assignment_loss‚3.016843926262211 1loss/FBt4/assignment_loss‚0.09416787175314312 0loss/SRqqt2/detection_loss‚2.549095033571638 1loss/FBt1/assignment_loss‚0.15950549288221996 0loss/FBt3/assignment_loss‚0.1180146139479778 .loss/FRt2/detection_loss‚2.384869142026071 &loss/total_loss‚41.741406440755945 _runtime‚7721.764668927 /loss/FBt2/detection_loss‚2.5364214594547567 /loss/FBt3/detection_loss‚0.5490173251695327 _step‚530 epoch‚5 1loss/SRqqt3/detection_loss‚0.8767239922617979 % _timestamp‚1.7885323382290947e+09 0loss/FRt3/assignment_loss‚1.2895768965993608 0loss/FRt4/assignment_loss‚3.5098980854858053 .loss/FRt1/detection_loss‚2.765980120861169 /loss/FRt3/detection_loss‚1.4576566404411717 1loss/SRqqt4/detection_loss‚2.1254074420684423 /loss/FBt1/detection_loss‚2.4709478744438718 .loss/FBt4/detection_loss‚3.203187962373098’Æ„ÿ”²‚(:‰ õ¬ëÔäÆóà gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.403' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.60075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X›?ºs³‚(:è õ¬ëÔ·¢á proc.cpu.threads‚17) disk./.usagePercent‚75.70100437530036% disk./.usageGB‚2707.4345932006836 disk.sda.out‚1311.7958984375 disk.nvme1n1.in‚ 19.140625 disk.md0p1.out‚ 1280.8984375 disk.sdb1.out‚0# disk.nvme0n1.out‚96044.11328125 disk.dm-0.in‚ 24.26953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 5.12890625! disk.dm-0.out‚122155.12109375$ memory_percent‚91.96112155914307) proc.memory.availableMB‚ 1317.08984375# proc.memory.rssMB‚ 2983.35546875 cpu‚25.22557098635073 disk.sda2.in‚128.625* proc.memory.percent‚18.208956718444824 disk.sdb1.in‚0 disk.sda1.out‚0! disk.sda2.out‚1311.7958984375 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda.in‚128.625 disk.md0.out‚ 1280.8984375 network.sent‚ 1.6573002e+07 disk.sdb2.in‚ 70.9609375 network.recv‚ 2.613348e+06" disk.nvme1n1.out‚ 26111.0078125 disk.md0p1.in‚ 199.5859375 disk.sdb.out‚1311.7958984375! disk.sdb2.out‚1311.7958984375 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875–3èö”´‚(:‰ „­ëÔ¹‡²à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îðRuµ‚(:ê „­ëÔ™¨ìà disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme1n1.out‚ 26111.3359375 disk.md0p1.out‚ 1282.35546875 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 5.12890625# disk.nvme0n1.out‚96331.52734375 disk.sdb1.in‚0 disk.md0p1.in‚ 199.5859375# proc.memory.rssMB‚ 2983.35546875 cpu‚25.22517023713382 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1282.35546875 disk.dm-0.in‚ 24.26953125 disk.sda.in‚128.625 disk.nvme1n1.in‚ 19.140625* proc.memory.percent‚18.208956718444824 network.sent‚ 1.6682825e+07) disk./.usagePercent‚75.69865165728281 disk.md0.in‚ 199.5859375 disk.sdb.out‚1313.3076171875 disk.sda.out‚1313.3076171875$ memory_percent‚91.96116924285889) proc.memory.availableMB‚ 1317.08203125 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚1313.3076171875 proc.cpu.threads‚17 network.recv‚ 2.618895e+06% disk./.usageGB‚2707.3504486083984 disk.sda1.in‚0! disk.dm-0.out‚122442.86328125 disk.sdb2.in‚ 70.9609375! disk.sda2.out‚1313.3076171875eFYI’¶‚(:‡ “­ëÔ—«µà gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•®?i·‚(:Þ “­ëÔÞÁóà disk.nvme1n1.out‚26111.5% disk./.usageGB‚2706.9044761657715 disk.sda2.in‚128.625# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 disk.nvme1n1.in‚ 19.140625 disk.md0p1.out‚ 1284.1015625# memory_percent‚91.9611930847168' proc.memory.availableMB‚ 1317.078125 cpu‚25.224799955491303 network.sent‚ 1.6686046e+07 network.recv‚ 2.622771e+06 disk.dm-0.in‚ 24.26953125! disk.dm-0.out‚122655.08203125! disk.sdb2.out‚1315.1181640625 disk.sda.in‚128.625 disk.md0.in‚ 199.5859375 disk.sdb1.out‚0 disk.sda1.out‚0' disk./.usagePercent‚75.686182081133 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1315.1181640625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 5.12890625 disk.sda.out‚1315.1181640625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0# disk.nvme0n1.out‚96543.58203125 disk.md0.out‚ 1284.1015625 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 2.466796875! disk.sda2.out‚1315.1181640625 proc.cpu.threads‚17ÂÀ÷|’¸‚(:‡ ¢­ëÔÕ«®à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿cœ˜q¹‚(:æ ¢­ëÔ©ßÇà% disk./.usageGB‚2706.8477935791016 disk.nvme0n1.in‚ 5.12890625 disk.sdb2.in‚ 70.9609375' proc.memory.availableMB‚ 1317.078125 disk.md0.out‚ 1285.6640625 cpu‚25.224433776518566) disk./.usagePercent‚75.68459721228619 disk.sdb1.in‚0 disk.dm-0.out‚122760.1328125 disk.rbd0.out‚ 2.466796875! disk.sda2.out‚1316.7470703125 disk.sda.out‚1316.7470703125# disk.nvme1n1.out‚26111.77734375# proc.memory.rssMB‚ 2983.35546875 disk.rbd2.in‚0 network.recv‚ 2.625835e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375# memory_percent‚91.9611930847168* proc.memory.percent‚18.208956718444824 disk.md0p1.in‚ 199.5859375# disk.nvme0n1.out‚96648.35546875 disk.sdb.out‚1316.7470703125! disk.sdb2.out‚1316.7470703125 disk.sda.in‚128.625 proc.cpu.threads‚17 disk.rbd2.out‚ 2.466796875 disk.md0p1.out‚ 1285.6640625 disk.sda2.in‚128.625 disk.sda1.in‚0 disk.dm-0.in‚ 24.26953125 disk.nvme1n1.in‚ 19.140625 disk.sda1.out‚0 network.sent‚ 1.6690838e+07Í9Ž¼2º- _wandbruntime‚7778  _runtime‚7778«+ ɻà ' lr-AdamW/pg1‚0.00038179409518988557 ' lr-AdamW/pg2‚0.00038179409518988557 trainer/global_step‚6499 % _timestamp‚1.7885323947276554e+09 _runtime‚7778.263135529 _step‚531y1QªÎ¼È ' lr-AdamW/pg1‚0.00038179409518988557 ' lr-AdamW/pg2‚0.00038179409518988557 trainer/global_step‚6499 % _timestamp‚1.7885323947276554e+09 _runtime‚7778.263135529 _step‚531“6øä¦2½- _wandbruntime‚7778  _runtime‚7778 ÿ‘ü'¾öO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135925734 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536938895317332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81441580 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚555085468697088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778890171392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611991491687936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529520439324144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135925732 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7446296305664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547640100365824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164612844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7417548018184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135925734 _runtime‚7778.264239934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚555085468697088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135925732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120901234 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604212372734464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547640082362880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7418455993188 trainer/global_step‚6499 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7446296305664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164639188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7445386334208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚555086396671488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779118953472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300538576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547640082362880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164612844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81441578 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300564922 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547640100365824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144¬o2(¿ûO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779118953472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300538576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547640082362880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164612844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81441578 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300564922 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547640100365824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135925734 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536938895317332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81441580 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚555085468697088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778890171392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611991491687936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529520439324144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135925732 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7446296305664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547640100365824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164612844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7417548018184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135925734 _runtime‚7778.264239934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚555085468697088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135925732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120901234 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604212372734464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547640082362880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7418455993188 trainer/global_step‚6499 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7446296305664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164639188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7445386334208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚555086396671488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152”•·2À- _wandbruntime‚7779  _runtime‚7779"ò˜Ç[ Á€N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚547721719451648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚537019030185362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300583567 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7446507794432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7419573266674 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81453645 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612082886806528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604302574498816 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7447417779200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537018105170790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164637675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120918957 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780053307392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚555169137230848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39465928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _step‚533 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604302598438912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529599456918688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚547721701448704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚555169137230848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300583567 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120919573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7418665291462 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164637675 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135945894 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚547721701448704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 _runtime‚7779.345857477 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300609939 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885323958092973e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300609939 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164664045 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81453645 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612082886806528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604302574498816 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7447417779200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537018105170790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164637675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120918957 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780053307392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚555169137230848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39465928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _step‚533 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604302598438912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529599456918688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚547721701448704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚555169137230848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300583567 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120919573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7418665291462 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164637675 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135945894 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚547721701448704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6499 _runtime‚7779.345857477 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300609939 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885323958092973e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300609939 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164664045 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551721054657024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165884090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165857746 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136933732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82044828 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551721036654080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7502383730688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121793707 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7501473759232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608723634016768 UADevicK‰ÍeStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚540944727285276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533471302011984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559222510413312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136933734 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7501473759232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7474333248296 _runtime‚7835.655455921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136933734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559223438387712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616561330993152 % _timestamp‚1.7885324521197805e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39748879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚533471319051344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7837673036288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551721054657024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82044830 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165884090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551721036654080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7473425273292 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165857746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192µÇë(ÓûO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7502383730688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559222510413312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚540945652299640 trainer/global_step‚6549 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551721054657024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165884090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165857746 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136933732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82044828 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551721036654080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7502383730688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121793707 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7501473759232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608723634016768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚540944727285276 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533471302011984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559222510413312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136933734 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7501473759232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7474333248296 _runtime‚7835.655455921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136933734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559223438387712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616561330993152 % _timestamp‚1.7885324521197805e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39748879 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚533471319051344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7837673036288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551721054657024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82044830 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165884090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551721036654080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7473425273292 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165857746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39749535 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570˜‡*22Ô- _wandbruntime‚7836  _runtime‚7836ΣÐà'Õ€N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165908947 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121811607 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608813859721216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7502595219456 _step‚537 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302862841 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551802655739904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302836469 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7838596642304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165882577 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82056893 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136953894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559306178947072 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551802655739904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136953894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7475450521782 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559305250959360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302862841 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121812226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885324530720363e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559306178947072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7503505204224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533550319606528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7503505204224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚616652456363520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165882577 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165908947 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¿0±€† Ö…N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302836469 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7474542546570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551802673742848 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121811607 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608813859721216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7502595219456 _step‚537 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302862841 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551802655739904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302836469 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7838596642304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165882577 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82056893 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136953894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559306178947072 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551802655739904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136953894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7475450521782 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559305250959360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302862841 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121812226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885324530720363e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559306178947072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7503505204224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533550319606528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatÙ.y‡…sMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7503505204224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚616652456363520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165882577 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165908947 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165908947 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537422198778544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613234943179264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 $ _timestamp‚1.788532509278583e+09 _runtime‚7892.814237023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7529302528400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82648080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137941734 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122686374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621130644857344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40038899 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 _step‚540 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555802008948224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚563360480103936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82648078 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7530210503404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167102648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305070726 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167128992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7558471155712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7557561184256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7895725618176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555801990945280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555802008948224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137941734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563359552129536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40038296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚544952409281948 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚563360480103936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7895939720192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122686979 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613234919239168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚6599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137941732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Q(çúO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137941734 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122686374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621130644857344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40038899 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 _step‚540 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555802008948224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚563360480103936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82648078 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7530210503404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167102648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305070726 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167128992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7558471155712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7557561184256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7895725618176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555801990945280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555802008948224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137941734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563359552129536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40038296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚544952409281948 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚563360480103936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7895939720192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122686979 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613234919239168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚6599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137941732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7558471155712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621130882899456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555801990945280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537422198778544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613234943179264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 $ _timestamp‚1.788532509278583e+09 _runtime‚7892.814237023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7529302528400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82648080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144œSú¶Ñ2è- _wandbruntime‚7893  _runtime‚7893¢7ʦé€N \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7559592629248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167127479 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7558682644480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚563442292675584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305115743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305089371 [MDeviceStatsMonitor.on_train_batch_end/inactivpÓK#e_split_bytes.small_pool.current‚ 222477312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7896905165824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545032544149978 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82660145 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137961892 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82660143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537501216373088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚541 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122704154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613325168883712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555883610031104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137961894 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7530419801678 _runtime‚7893.794487781 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305089371 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚563443220663296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885325102583306e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚563443220663296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚555883628034048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621222050109440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885325102583306e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚563443220663296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚555883628034048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621222050109440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305115743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305089371 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7896905165824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545032544149978 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82660145 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137961892 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82660143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537501216373088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚541 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122704154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613325168883712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393'/›2ë- _wandbruntime‚7893  _runtime‚7893¯ EÜcìÝ 1loss/FBt1/assignment_loss‚0.15525531167770623 1loss/FBt2/assignment_loss‚0.05350928119150211 1loss/SRqqt3/detection_loss‚0.9373170665154854 .loss/FBt4/detection_loss‚3.139814088741938 epoch‚5 0loss/FRt2/assignment_loss‚3.2028372235920117 2loss/SRqqt1/assignment_loss‚1.3039488396767913 /loss/FBt3/detection_loss‚0.5326267945301997 0loss/FRt1/assignment_loss‚2.1931872831450567 2loss/SRqqt2/assignment_loss‚1.4999778183961385 2loss/SRqqt4/assignment_loss‚1.5440074204984637 2loss/FBt3/assignment_loss‚0.049165142965986015 /loss/FRt3/detection_loss‚1.3052097685061967 /loss/FBt1/detection_loss‚2.5446226447820663 1loss/SRqqt4/detection_loss‚1.9929731435245939 trainer/global_step‚6599 _step‚542 &loss/total_loss‚41.728937507576475 1loss/FBt4/assignment_loss‚0.03856517141154351 /loss/FRt1/detection_loss‚3.0747404054359153 /loss/FRt2/detection_loss‚2.9235987103557934 /loss/FBt2/detection_loss‚2.1760617569088936 % _timestamp‚1.7885325102601557e+09 _runtime‚7893.795471946 0loss/FRt4/assignment_loss‚3.8144174461011535 /loss/FRt4/detection_loss‚2.0094979769653745 1loss/SRqqt2/detection_loss‚2.0862686038017273 0loss/FRt3/assignment_loss‚1.5723899795458867 1loss/SRqqt1/detection_loss‚2.7061206883397597 1loss/SRqqt3/assignment_loss‚0.872824940966287¢—Ghíâ 0loss/FRt1/assignment_loss‚2.1931872831450567 2loss/SRqqt2/assignment_loss‚1.4999778183961385 2loss/SRqqt4/assignment_loss‚1.5440074204984637 2loss/FBt3/assignment_loss‚0.049165142965986015 /loss/FRt3/detection_loss‚1.3052097685061967 /loss/FBt1/detection_loss‚2.5446226447820663 1loss/SRqqt4/detection_loss‚1.9929731435245939 trainer/global_step‚6599 _step‚542 &loss/total_loss‚41.728937507576475 1loss/FBt4/assignment_loss‚0.03856517141154351 /loss/FRt1/detection_loss‚3.0747404054359153 /loss/FRt2/detection_loss‚2.9235987103557934 /loss/FBt2/detection_loss‚2.1760617569088936 % _timestamp‚1.7885325102601557e+09 _runtime‚7893.795471946 0loss/FRt4/assignment_loss‚3.8144174461011535 /loss/FRt4/detection_loss‚2.0094979769653745 1loss/SRqqt2/detection_loss‚2.0862686038017273 0loss/FRt3/assignment_loss‚1.5723899795458867 1loss/SRqqt1/detection_loss‚2.7061206883397597 1loss/SRqqt3/assignment_loss‚0.872824940966287 1loss/FBt1/assignment_loss‚0.15525531167770623 1loss/FBt2/assignment_loss‚0.05350928119150211 1loss/SRqqt3/detection_loss‚0.9373170665154854 .loss/FBt4/detection_loss‚3.139814088741938 epoch‚5 0loss/FRt2/assignment_loss‚3.2028372235920117 2loss/SRqqt1/assignment_loss‚1.3039488396767913 /loss/FBt3/detection_loss‚0.5326267945301997žâÇÿ”î‚(:‰ ©®ëÔ’ÇŸà gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò(mï‚(:â ©®ëÔççÙá disk.sda1.in‚0$ memory_percent‚91.96207523345947) proc.memory.availableMB‚ 1316.93359375* proc.memory.percent‚18.208956718444824" disk.sda2.out‚1331.79931640625 disk.sda.in‚128.625 network.sent‚ 1.6959814e+07( disk./.usagePercent‚75.6861907206361 proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 5.26171875# proc.memory.rssMB‚ 2983.35546875" disk.sdb2.out‚1331.79931640625 disk.md0p1.out‚ 1300.1875 disk.md0.out‚ 1300.1875 cpu‚25.2209832629283 disk.sda2.in‚128.625! disk.dm-0.out‚125058.56640625! disk.sda.out‚1331.79931640625 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875# disk.nvme0n1.out‚98944.46484375 disk.md0.in‚ 199.5859375 disk.nvme1n1.in‚ 22.046875 network.recv‚ 2.666639e+06# disk./.usageGB‚2706.90478515625 disk.dm-0.in‚ 27.30859375 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb.out‚1331.79931640625 disk.md0p1.in‚ 199.5859375 disk.sdb1.out‚0 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 26114.1015625Èáo”ð‚(:‰ ¸®ëÔ뙫à gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.137' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.53425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c:~wñ‚(:ì ¸®ëÔŒ‚ÿà disk.sda.in‚128.625! disk.nvme1n1.out‚ 26114.984375 disk.md0p1.out‚ 1301.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚125216.76953125) proc.memory.availableMB‚ 1316.68359375# disk./.usageGB‚2706.84814453125 disk.sdb2.in‚ 70.9609375$ memory_percent‚91.96360111236572 proc.cpu.threads‚17 network.sent‚ 1.7056741e+07 disk.rbd0.in‚0# disk.nvme0n1.out‚99101.78515625! disk.sdb.out‚1333.19970703125! disk.sda.out‚1333.19970703125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 5.26171875 disk.md0.in‚ 199.5859375 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625 disk.md0.out‚ 1301.52734375 cpu‚25.220633666260735) disk./.usagePercent‚75.68460702505516" disk.sda2.out‚1333.19970703125 disk.nvme1n1.in‚ 22.046875 disk.rbd0.out‚ 2.466796875 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 27.30859375 network.recv‚ 2.671983e+06 disk.md0p1.in‚ 199.5859375" disk.sdb2.out‚1333.19970703125# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 ­|ñ‘ò‚(:† Ç®ëÔ»ûÌà gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M·ró‚(:ç Ç®ëԀ܋á disk.sda2.in‚128.625 network.sent‚ 1.7059927e+07" disk.nvme1n1.out‚ 26201.4140625 disk.rbd2.out‚ 2.466796875$ memory_percent‚91.96207523345947 proc.cpu.threads‚17 disk.md0.in‚ 199.5859375 disk.dm-0.out‚ 125460.4375! disk.sdb.out‚1334.66748046875! disk.sda.out‚1334.66748046875 disk.md0p1.out‚ 1302.9453125) proc.memory.availableMB‚ 1316.93359375 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0* proc.memory.percent‚18.208956718444824 disk.nvme1n1.in‚ 22.046875 disk.nvme0n1.in‚ 5.26171875% disk./.usageGB‚2706.9323120117188 disk.md0.out‚ 1302.9453125 cpu‚25.220248621246693 disk.dm-0.in‚ 27.30859375 disk.sda.in‚128.625 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 99259.0234375# proc.memory.rssMB‚ 2983.35546875 network.recv‚ 2.676116e+06( disk./.usagePercent‚75.6869603830359 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sda2.out‚1334.66748046875" disk.sdb2.out‚1334.66748046875 disk.sda1.out‚0vJH2ô- _wandbruntime‚7949  _runtime‚7949ŸÉ8Äõ¾ % lr-AdamW/pg1‚0.000365952559261768 % lr-AdamW/pg2‚0.000365952559261768 trainer/global_step‚6649 $ _timestamp‚1.788532565700783e+09 _runtime‚7949.236242992 _step‚543㥠Éöà $ _timestamp‚1.788532565700783e+09 _runtime‚7949.236242992 _step‚543 % lr-AdamW/pg1‚0.000365952559261768 % lr-AdamW/pg2‚0.000365952559261768 trainer/global_step‚6649Ÿ=ÑN2÷- _wandbruntime‚7949  _runtime‚7949J±6 ƒøõO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7613648609280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559882945236480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83251328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559882945236480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625700869323264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307323628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123579930 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138949734 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚567497521820160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83251330 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307297282 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7614558580736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328602 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7954410042368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788532565701767e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168373894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617746228401664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚7949.237349648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138949732 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168347550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559882963239424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625700614503936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7954640921600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7585179783508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_st‰â¿xart/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617746204461568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚6649 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567496593845760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚541373061466384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7614558580736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚567497521820160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚541373078505744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138949734 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548958241249892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123580565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40329235 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7954410042368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788532565701767e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168373894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617746228401664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚7949.237349648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138949732 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168347550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559882963239424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625700614503936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7954640921600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7585179783508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617746204461568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚6649 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567496593845760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚541373061466384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7614558580736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚567497521820160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚541373078505744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138949734 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548958241249892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123580565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40329235 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123597712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83263393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7614770069504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549038376117714 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541452079060928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138969894 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617836454106112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83263395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7614770069504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138969892 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617836430166016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚549039301132286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307368645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7955575613440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚6649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚567579334391808 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138969892 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885325666419096e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚567580262379520 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168398751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0á?jþ…N BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123597712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83263393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7614770069504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549038376117714 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541452079060928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138969894 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617836454106112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83263395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7614770069504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138969892 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617836430166016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚549039301132286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_qmßð end/allocation.all.allocated‚ 307368645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7955575613440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚6649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚567579334391808 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138969892 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885325666419096e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚567580262379520 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168398751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559964582325248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¡“·—è2ÿ- _wandbruntime‚7950  _runtime‚7950K ÅX€Ò _runtime‚7950.179570809 /loss/FRt4/assignment_loss‚2.562746082411872 0loss/FBt2/assignment_loss‚0.1056106652393396 /loss/FRt1/detection_loss‚3.1102636456489563 .loss/FRt4/detection_loss‚2.245451335774528 1loss/SRqqt4/detection_loss‚1.7248697025435311 _step‚546 .loss/FRt1/assignment_loss‚4.10240793639216 2loss/SRqqt3/assignment_loss‚1.0428834008463823 .loss/FRt2/detection_loss‚2.559768333545435 /loss/FRt3/detection_loss‚1.4838977543198888 1loss/SRqqt3/detection_loss‚0.8964761825753194 1loss/SRqqt1/assignment_loss‚1.362424100852675 /loss/FBt1/detection_loss‚2.5375501066446304 /loss/FBt4/detection_loss‚2.5979986131191253 epoch‚5 %loss/total_loss‚42.87207358692747 % _timestamp‚1.7885325666442916e+09 0loss/FRt2/assignment_loss‚3.1125740549143623 1loss/FBt4/assignment_loss‚0.45504718210759165 /loss/FBt3/detection_loss‚0.6677292872191174 /loss/FRt3/assignment_loss‚1.143178584821084 2loss/SRqqt2/assignment_loss‚1.3953241167160182 1loss/FBt1/assignment_loss‚0.02320264394802507 0loss/SRqqt1/detection_loss‚3.287576844294866 trainer/global_step‚6649 2loss/SRqqt4/assignment_loss‚1.4144181444363702 1loss/FBt3/assignment_loss‚0.10452291127354374 0loss/SRqqt2/detection_loss‚2.211650034556022 .loss/FBt2/detection_loss‚2.724501922726631;–Ø™]× 1loss/SRqqt1/assignment_loss‚1.362424100852675 /loss/FBt1/detection_loss‚2.5375501066446304 /loss/FBt4/detection_loss‚2.5979986131191253 epoch‚5 %loss/total_loss‚42.87207358692747 % _timestamp‚1.7885325666442916e+09 0loss/FRt2/assignment_loss‚3.1125740549143623 1loss/FBt4/assignment_loss‚0.45504718210759165 /loss/FBt3/detection_loss‚0.6677292872191174 /loss/FRt3/assignment_loss‚1.143178584821084 2loss/SRqqt2/assignment_loss‚1.3953241167160182 1loss/FBt1/assignment_loss‚0.02320264394802507 0loss/SRqqt1/detection_loss‚3.287576844294866 trainer/global_step‚6649 2loss/SRqqt4/assignment_loss‚1.4144181444363702 1loss/FBt3/assignment_loss‚0.10452291127354374 0loss/SRqqt2/detection_loss‚2.211650034556022 .loss/FBt2/detection_loss‚2.724501922726631 _runtime‚7950.179570809 /loss/FRt4/assignment_loss‚2.562746082411872 0loss/FBt2/assignment_loss‚0.1056106652393396 /loss/FRt1/detection_loss‚3.1102636456489563 .loss/FRt4/detection_loss‚2.245451335774528 1loss/SRqqt4/detection_loss‚1.7248697025435311 _step‚546 .loss/FRt1/assignment_loss‚4.10240793639216 2loss/SRqqt3/assignment_loss‚1.0428834008463823 .loss/FRt2/detection_loss‚2.559768333545435 /loss/FRt3/detection_loss‚1.4838977543198888 1loss/SRqqt3/detection_loss‚0.8964761825753194¢k©Ã ’‚‚(:‡ å®ëÔÜÒß gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“×._rƒ‚(:ç å®ëԤݖà network.recv‚ 2.685956e+06) disk./.usagePercent‚75.66392714778473 disk.nvme1n1.in‚ 22.5234375 disk.sdb.in‚ 70.9609375# proc.memory.rssMB‚ 2983.35546875 proc.cpu.threads‚17% disk./.usageGB‚2706.1085319519043 disk.rbd0.in‚0 disk.sdb1.in‚0$ memory_percent‚91.96217060089111# disk.nvme1n1.out‚26302.23046875 disk.sdb1.out‚0 disk.nvme0n1.in‚6.234375 disk.dm-0.out‚125735.9921875" disk.sdb2.out‚1338.10400390625 disk.md0p1.out‚ 1306.28125 disk.rbd0.out‚ 2.466796875 disk.sda1.in‚0" disk.sda2.out‚1338.10400390625 disk.dm-0.in‚ 28.7578125 disk.sdb2.in‚ 70.9609375) proc.memory.availableMB‚ 1316.91796875 network.sent‚ 1.7162128e+07 disk.rbd2.out‚ 2.466796875 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚99433.76171875 disk.sda2.in‚128.625 disk.md0.out‚ 1306.28125! disk.sda.out‚1338.10400390625* proc.memory.percent‚18.208956718444824 cpu‚25.219458923038797 disk.md0p1.in‚ 199.5859375! disk.sdb.out‚1338.10400390625£ÒQò“„‚(:ˆ ô®ëÔª«ÞÞ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w9çr…‚(:ç ô®ëÔâÒŒà disk.sdb.in‚ 70.9609375! disk.sdb.out‚1339.57470703125 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 disk.md0.in‚ 199.5859375 disk.sda.in‚128.625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚128.625" disk.sda2.out‚1339.57470703125 disk.md0.out‚ 1307.69921875 disk.nvme1n1.in‚ 22.5234375 cpu‚25.218985623570344 disk.md0p1.in‚ 199.5859375) proc.memory.availableMB‚ 1316.62109375* proc.memory.percent‚18.208956718444824) disk./.usagePercent‚75.67280290396525 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 28.7578125 disk.md0p1.out‚ 1307.69921875$ memory_percent‚91.96398258209229 network.recv‚ 2.691261e+06% disk./.usageGB‚2706.4259719848633! disk.nvme0n1.out‚ 99503.703125 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875 disk.dm-0.out‚ 126001.734375 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1339.57470703125 disk.nvme1n1.out‚ 26498.03125# proc.memory.rssMB‚ 2983.35546875 disk.sda1.in‚0" disk.sdb2.out‚1339.57470703125 network.sent‚ 1.7261493e+07 disk.nvme0n1.in‚6.234375Ó‰”†‚(:‰ ƒ¯ëÔÓ©à gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðj¿Åm‡‚(:â ƒ¯ëÔ¥¢¾á disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚6.234375 proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1309.1640625! disk.dm-0.out‚126103.84765625 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚26595.80859375# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 network.sent‚ 1.7264655e+07% disk./.usageGB‚2706.4301986694336 disk.rbd2.in‚0 disk.sdb2.out‚1341.091796875 disk.md0p1.in‚ 199.5859375( proc.memory.availableMB‚ 1316.7421875 cpu‚25.218473423717775 disk.md0.out‚ 1309.1640625 disk.sda.in‚128.625 disk.sda.out‚1341.091796875 disk.sdb1.out‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.sda1.out‚0$ memory_percent‚91.96324348449707 disk.sdb.out‚1341.091796875 disk.nvme1n1.in‚ 22.5234375 network.recv‚ 2.695104e+06) disk./.usagePercent‚75.67292108383484" disk.nvme0n1.out‚ 99508.0390625 disk.sda2.out‚1341.091796875 disk.dm-0.in‚ 28.7578125 disk.rbd0.in‚0UÜé2ˆ- _wandbruntime‚8002  _runtime‚8002Æe–OljÁ & lr-AdamW/pg1‚0.0003606705576684313 & lr-AdamW/pg2‚0.0003606705576684313 trainer/global_step‚6699 % _timestamp‚1.7885326184730651e+09 _runtime‚8002.008671183 _step‚547¸~¯ÌŠÆ & lr-AdamW/pg1‚0.0003606705576684313 & lr-AdamW/pg2‚0.0003606705576684313 trainer/global_step‚6699 % _timestamp‚1.7885326184730651e+09 _runtime‚8002.008671183 _step‚547£`"E£2‹- _wandbruntime‚8002  _runtime‚8002ŸÎ/ü'ŒöO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7641057038616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169618796 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169618796 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7641965013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40619441 _runtime‚8002.009775938 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571634563536384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83854578 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8013006722048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563963899527680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139957734 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139957732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169592452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545323958232944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670646005760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630270757516800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169592452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139957732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8013243892736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚552965923246564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670646005760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571633635561984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139957734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563963917530624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7669736034304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309576530 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630270496406016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622257513624064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563963917530624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7669736034304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622257489683968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83854580 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169618796 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7641965013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40619441 _runtime‚8002.009775938 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571634563536384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83854578 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8013006722048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563963899527680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139957734 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139957732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169592452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545323958232944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670646005760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630270757516800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169592452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139957732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8013243892736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚552965923246564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670646005760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571633635561984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139957734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563963917530624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7669736034304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309576530 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630270496406016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622257513624064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563963917530624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7669736034304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622257489683968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456¤,°Žn2Ž- _wandbruntime‚8003  _runtime‚80033k´{€N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309595175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545402958788128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚630361889189888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current ´ |‹$‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7642174311894 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169643653 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚571717304095744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622347739328512 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139977892 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622347715388416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83866645 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545402975827488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309621547 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671767479296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169617283 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139977894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169643653 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7670857494528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚571717304095744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚564045518613504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169617283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚6699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40625250 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚564045518613504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309621547 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚8003.016929021 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630362141898752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124491275 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7670857494528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚571717304095744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚564045518613504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169617283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚6699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40625250 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚564045518613504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309621547 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚8003.016929021 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630362141898752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124491275 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622347715388416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83866645 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545402975827488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309621547 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671767479296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169617283 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¥´À¼M2‘- _wandbruntime‚8003  _runtime‚8003RÊšì[’Õ 0loss/SRqqt2/detection_loss‚1.758000430908609 0loss/FRt2/assignment_loss‚3.5837268911559006 0loss/FBt4/assignment_loss‚0.5403019374290455 0loss/SRqqt3/detection_loss‚0.996290123882763 _runtime‚8003.018461768 /loss/FRt3/assignment_loss‚1.167073392261893 &loss/total_loss‚40.368549744785106 /loss/FRt1/assignment_loss‚3.261452661620246 2loss/SRqqt1/assignment_loss‚1.8402611660421826 0loss/SRqqt4/detection_loss‚2.071794177426232 0loss/FBt3/detection_loss‚0.48827015242672395 epoch‚5 0loss/FBt2/assignment_loss‚0.1414392024376413 2loss/FBt3/assignment_loss‚0.036285456565529166 /loss/FRt4/detection_loss‚2.0475548968281387 /loss/FBt4/detection_loss‚2.2366627617315813 _step‚550 .loss/FRt2/detection_loss‚2.572682333861692 % _timestamp‚1.7885326194830506e+09 0loss/FRt4/assignment_loss‚2.2455682732024282 2loss/SRqqt4/assignment_loss‚1.4120785368813409 1loss/FBt1/assignment_loss‚0.29040056405942777 1loss/SRqqt1/detection_loss‚2.1888693794608116 .loss/FBt1/detection_loss‚2.968380892276764 /loss/FBt2/detection_loss‚1.6566390301051892 trainer/global_step‚6699 1loss/SRqqt2/assignment_loss‚1.315356157720089 2loss/SRqqt3/assignment_loss‚1.1279737102462257 .loss/FRt1/detection_loss‚2.879404151881183 /loss/FRt3/detection_loss‚1.5420834643734713} f~`“Ú 1loss/SRqqt2/assignment_loss‚1.315356157720089 2loss/SRqqt3/assignment_loss‚1.1279737102462257 .loss/FRt1/detection_loss‚2.879404151881183 /loss/FRt3/detection_loss‚1.5420834643734713 0loss/SRqqt2/detection_loss‚1.758000430908609 0loss/FRt2/assignment_loss‚3.5837268911559006 0loss/FBt4/assignment_loss‚0.5403019374290455 0loss/SRqqt3/detection_loss‚0.996290123882763 _runtime‚8003.018461768 /loss/FRt3/assignment_loss‚1.167073392261893 &loss/total_loss‚40.368549744785106 /loss/FRt1/assignment_loss‚3.261452661620246 2loss/SRqqt1/assignment_loss‚1.8402611660421826 0loss/SRqqt4/detection_loss‚2.071794177426232 0loss/FBt3/detection_loss‚0.48827015242672395 epoch‚5 0loss/FBt2/assignment_loss‚0.1414392024376413 2loss/FBt3/assignment_loss‚0.036285456565529166 /loss/FRt4/detection_loss‚2.0475548968281387 /loss/FBt4/detection_loss‚2.2366627617315813 _step‚550 .loss/FRt2/detection_loss‚2.572682333861692 % _timestamp‚1.7885326194830506e+09 0loss/FRt4/assignment_loss‚2.2455682732024282 2loss/SRqqt4/assignment_loss‚1.4120785368813409 1loss/FBt1/assignment_loss‚0.29040056405942777 1loss/SRqqt1/detection_loss‚2.1888693794608116 .loss/FBt1/detection_loss‚2.968380892276764 /loss/FBt2/detection_loss‚1.6566390301051892 trainer/global_step‚6699¦´þ””‚(:‰ ’¯ëÔখß gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£O|d•‚(:Ù ’¯ëÔ®Õ»ß disk.sda2.in‚128.625 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚1342.5244140625 disk.rbd2.out‚ 2.466796875 disk.sda.out‚1342.5244140625 disk.sda1.in‚0& proc.memory.availableMB‚ 1316.34375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0# memory_percent‚91.9656753540039 network.sent‚ 1.7270258e+07 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.out‚ 99508.40625 disk.md0.out‚ 1310.546875 disk.sdb.out‚1342.5244140625 disk.nvme0n1.in‚6.234375 cpu‚25.21795422398102 network.recv‚ 2.698722e+06) disk./.usagePercent‚75.67467436966186 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 28.7578125! disk.nvme1n1.out‚ 26693.828125 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.sda2.out‚1342.5244140625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.nvme1n1.in‚ 22.5234375 disk.md0p1.out‚ 1310.546875 disk.sdb1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚18.208956718444824 disk.dm-0.out‚ 126202.234375 disk.sda1.out‚0# proc.memory.rssMB‚ 2983.35546875$ disk./.usageGB‚2706.492904663086 §·’–‚(:‡ ¡¯ëÔ•ƒ¥à gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ŬY—‚(:Î ¡¯ëÔÜšÊà disk.rbd2.in‚0 disk.md0.out‚ 1312.64453125! disk.dm-0.out‚126351.80859375 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.sda1.out‚0 disk.sda.out‚ 1344.6875 disk.nvme1n1.in‚ 22.5234375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875) proc.memory.availableMB‚ 1316.16796875 cpu‚25.21738543608337 disk.sda1.in‚0 disk.sda2.in‚128.625 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 28.7578125 disk.sda.in‚128.625 network.recv‚ 2.704404e+06$ disk./.usageGB‚2706.555404663086! disk.nvme0n1.out‚ 99560.234375 disk.sda2.out‚ 1344.6875# disk.nvme1n1.out‚26791.57421875) disk./.usagePercent‚75.67642189582014 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚6.234375 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0$ memory_percent‚91.96674823760986 disk.md0p1.out‚ 1312.64453125# proc.memory.rssMB‚ 2983.35546875* proc.memory.percent‚18.208956718444824 network.sent‚ 1.7377872e+07 disk.sdb.out‚ 1344.6875 disk.sdb2.out‚ 1344.6875 disk.sdb1.in‚0 proc.cpu.threads‚17 ‚”˜‚(:‰ °¯ëÔíñ€ß gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e%Ýq™‚(:æ °¯ëÔÉ·¿à network.recv‚ 2.707899e+06% disk./.usageGB‚2706.6801376342773 disk.md0.out‚ 1314.09765625 disk.sda2.in‚128.625" disk.sdb2.out‚1346.19873046875 disk.nvme1n1.in‚ 22.5234375 disk.md0p1.in‚ 199.5859375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875$ memory_percent‚91.96455478668213# proc.memory.rssMB‚ 2983.35546875 disk.md0.in‚ 199.5859375 disk.nvme0n1.in‚6.234375 cpu‚25.21688874148697 proc.cpu.threads‚17! disk.nvme0n1.out‚ 99664.109375) disk./.usagePercent‚75.67990948189946 disk.dm-0.out‚ 126553.53125 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1314.09765625 disk.rbd2.in‚0 network.sent‚ 1.7383093e+07" disk.sda2.out‚1346.19873046875 disk.dm-0.in‚ 28.7578125 disk.sda1.out‚0) proc.memory.availableMB‚ 1316.52734375* proc.memory.percent‚18.208956718444824! disk.sdb.out‚1346.19873046875 disk.sda.in‚128.625! disk.sda.out‚1346.19873046875! disk.nvme1n1.out‚ 26889.421875 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875Žš¶)2š- _wandbruntime‚8054  _runtime‚8054çL€.Æ›À & lr-AdamW/pg1‚0.0003553896458283266 & lr-AdamW/pg2‚0.0003553896458283266 trainer/global_step‚6749 $ _timestamp‚1.788532671297521e+09 _runtime‚8054.833101569 _step‚551#½Åa˜Š& lr-AdamW/pg1‚0.0003553896458283266 & lr-AdamW/pg2‚0.0003553896458283266 trainer/global_step‚6749 $ _timestamp‚1.788532671297521e+09 _runtime‚8054.833101569 _step‚551§u[Ü32- _wandbruntime‚8054  _runtime‚8054¤æ†š0žöO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863698 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚575771605252608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170837354 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575770677278208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556971755214508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568044853818880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7725823459328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40910032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7696934293724 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚634840797427200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8071763507200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140965734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556972680228872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7726733430784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549274837960144 trainer/global_step‚6749 _step‚552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7726733430784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚8054.834317154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140965732 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549274820920784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311803086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568044871821824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144*ºÝ(ŸûO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575770677278208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556971755214508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568044853818880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7725823459328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40910032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7696934293724 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚634840797427200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8071763507200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140965734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556972680228872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7726733430784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549274837960144 trainer/global_step‚6749 _step‚552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7726733430784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚8054.834317154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140965732 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549274820920784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311803086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568044871821824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863698 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚575771605252608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170837354 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393¨÷`©Ï” ‚(:‰ ¿¯ëÔÌÿß gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°)í˜p¡‚(:å ¿¯ëÔóïÉà$ disk./.usageGB‚2706.400043487549 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.5859375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚6.234375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 28.7578125 disk.dm-0.out‚ 126712.1875 disk.sdb.in‚ 70.9609375 disk.sda2.in‚128.625) proc.memory.availableMB‚ 1316.65234375 disk.md0p1.out‚ 1315.50390625 disk.rbd0.out‚ 2.466796875# proc.memory.rssMB‚ 2983.35546875 cpu‚25.2163616779932) disk./.usagePercent‚75.67207793232842 disk.nvme1n1.in‚ 22.5234375 disk.sdb1.out‚0 disk.sda1.out‚0! disk.sdb.out‚1347.66357421875" disk.sdb2.out‚1347.66357421875! disk.sda.out‚1347.66357421875 disk.md0.in‚ 199.5859375 disk.md0.out‚ 1315.50390625 network.sent‚ 1.7388465e+07 network.recv‚ 2.711533e+06 disk.sda.in‚128.625 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sda2.out‚1347.66357421875" memory_percent‚91.963791847229# disk.nvme1n1.out‚26944.41796875# disk.nvme0n1.out‚99767.76953125* proc.memory.percent‚18.208956718444824†ž€è2¢- _wandbruntime‚8055  _runtime‚8055äà¯'£€N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚575853417824256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚553 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40915216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626859024550912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557052815096902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170888555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311874449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40915834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7726944919552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568126490907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549353838515328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7727854904320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84469893 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚557051890082330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626859000610816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170888555 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170862185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8073156853760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7698051567002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125385109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311848077 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7727854904320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311848077 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140985894 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568126472904704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140985892 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634932181404672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚634931926598656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885326723079531e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7726944919552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚575854345811968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚575854345811968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8072925987840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚568126490907648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568126472904704 _runtime‚8055.844531501 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311874449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549353855554688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0÷Ÿñ»Æ¤…N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125385109 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311848077 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7727854904320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311848077 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140985894 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568126472904704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140985892 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634932181404672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚634931926598656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885326723079531e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7726944919552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚575854345811968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚575854345811968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8072925987840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚568126490907648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568126472904704 _runtime‚8055.844531501 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311874449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549353855554688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚575853417824256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚553 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40915216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626859024550912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557052815096902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170888555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311874449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40915834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7726944919552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568126490907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549353838515328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7727854904320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84469893 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚557051890082330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626859000610816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170888555 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170862185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8073156853760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7698051567002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784©}_êò2¥- _wandbruntime‚8055  _runtime‚8055ºä²Y¦Ó /loss/FRt1/detection_loss‚2.7648178398609162 .loss/FRt3/detection_loss‚1.369635211614271 /loss/FBt3/detection_loss‚0.5760081922169775 1loss/SRqqt3/detection_loss‚0.7879703289286776 .loss/FBt1/detection_loss‚2.921309437070574 1loss/FBt1/assignment_loss‚0.04793854414765519 .loss/FRt2/detection_loss‚2.326845528337109 .loss/FBt2/detection_loss‚5.152423664927483 % _timestamp‚1.7885326723102837e+09 /loss/FRt4/assignment_loss‚2.853433144398225 1loss/FBt3/assignment_loss‚0.05056119914787347 1loss/FBt4/assignment_loss‚0.09475705594559258 /loss/FBt4/detection_loss‚2.4618822173638777 _runtime‚8055.845666566 1loss/SRqqt1/assignment_loss‚1.783368629488078 2loss/SRqqt3/assignment_loss‚0.9533258356399589 2loss/SRqqt4/assignment_loss‚1.4080702596558974 /loss/FRt4/detection_loss‚2.3749471075164204 1loss/SRqqt1/detection_loss‚2.5176665259130075 %loss/total_loss‚44.16245471623718 epoch‚5 _step‚554 0loss/FRt2/assignment_loss‚3.4489108899544023 0loss/SRqqt2/detection_loss‚2.015630831772631 0loss/SRqqt4/detection_loss‚2.042985199353634 3loss/FBt2/assignment_loss‚0.0017869466901014874 trainer/global_step‚6749 0loss/FRt1/assignment_loss‚3.6197252531846362 .loss/FRt3/assignment_loss‚1.10350488225619 0loss/SRqqt2/assignment_loss‚1.48494999085299ñ¦5q^§Ø trainer/global_step‚6749 0loss/FRt1/assignment_loss‚3.6197252531846362 .loss/FRt3/assignment_loss‚1.10350488225619 0loss/SRqqt2/assignment_loss‚1.48494999085299 /loss/FRt1/detection_loss‚2.7648178398609162 .loss/FRt3/detection_loss‚1.369635211614271 /loss/FBt3/detection_loss‚0.5760081922169775 1loss/SRqqt3/detection_loss‚0.7879703289286776 .loss/FBt1/detection_loss‚2.921309437070574 1loss/FBt1/assignment_loss‚0.04793854414765519 .loss/FRt2/detection_loss‚2.326845528337109 .loss/FBt2/detection_loss‚5.152423664927483 % _timestamp‚1.7885326723102837e+09 /loss/FRt4/assignment_loss‚2.853433144398225 1loss/FBt3/assignment_loss‚0.05056119914787347 1loss/FBt4/assignment_loss‚0.09475705594559258 /loss/FBt4/detection_loss‚2.4618822173638777 _runtime‚8055.845666566 1loss/SRqqt1/assignment_loss‚1.783368629488078 2loss/SRqqt3/assignment_loss‚0.9533258356399589 2loss/SRqqt4/assignment_loss‚1.4080702596558974 /loss/FRt4/detection_loss‚2.3749471075164204 1loss/SRqqt1/detection_loss‚2.5176665259130075 %loss/total_loss‚44.16245471623718 epoch‚5 _step‚554 0loss/FRt2/assignment_loss‚3.4489108899544023 0loss/SRqqt2/detection_loss‚2.015630831772631 0loss/SRqqt4/detection_loss‚2.042985199353634 3loss/FBt2/assignment_loss‚0.0017869466901014874ªŸ”CŽ”¨‚(:‰ ίëÔŦ¥â gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¬2r©‚(:ç ίëÔ÷…ã network.recv‚ 2.716505e+06 disk.sdb1.in‚0 disk.nvme0n1.in‚6.234375! disk.sdb.out‚1349.55126953125! disk.sda.out‚1349.55126953125# disk./.usageGB‚2706.34334564209 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚128.625 disk.md0.out‚ 1317.33984375$ memory_percent‚91.96646213531494* proc.memory.percent‚18.208956718444824 network.sent‚ 1.7484388e+07( disk./.usagePercent‚75.6704926368395 disk.rbd0.out‚ 2.466796875! disk.nvme0n1.out‚ 99871.578125 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 22.5234375 disk.md0p1.in‚ 199.5859375" disk.sda2.out‚1349.55126953125 disk.md0.in‚ 199.5859375 disk.dm-0.out‚126817.0078125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 28.7578125 disk.md0p1.out‚ 1317.33984375" disk.nvme1n1.out‚ 26945.4296875 disk.rbd0.in‚0 disk.rbd2.out‚ 2.466796875" disk.sdb2.out‚1349.55126953125 cpu‚25.215853785935384 disk.sda.in‚128.625) proc.memory.availableMB‚ 1316.21484375# proc.memory.rssMB‚ 2983.35546875$u䔪‚(:‰ ݯëÔ×·Ùà gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O,Íýu«‚(:ê ݯëÔŒ°íá network.sent‚ 1.7581577e+07! disk.dm-0.out‚127007.05078125" disk.sdb2.out‚1351.01123046875! disk.sdb.out‚1351.01123046875 network.recv‚ 2.722196e+06) disk./.usagePercent‚75.67284706142554% disk./.usageGB‚2706.4275512695312 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 1318.75390625 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0" disk.sda2.out‚1351.01123046875 disk.nvme1n1.in‚ 22.5234375 disk.nvme1n1.out‚ 26945.5625) proc.memory.availableMB‚ 1316.72265625* proc.memory.percent‚18.208956718444824 cpu‚25.21539877930005 proc.cpu.threads‚17$ disk.nvme0n1.out‚100061.48828125 disk.sda1.in‚0 disk.nvme0n1.in‚6.234375$ memory_percent‚91.96336269378662! disk.sda.out‚1351.01123046875 disk.md0p1.in‚ 199.5859375 disk.rbd0.out‚ 2.466796875 disk.md0.in‚ 199.5859375# proc.memory.rssMB‚ 2983.35546875 disk.md0p1.out‚ 1318.75390625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 28.7578125 disk.rbd2.out‚ 2.466796875 disk.sda2.in‚128.625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625ŒÊØ㑬‚(:† ì¯ëÔѬûÞ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01X5j­‚(:ß ì¯ëÔ¤®ˆâ network.recv‚ 2.726461e+06 disk.sda1.out‚0 disk.md0.in‚ 199.5859375 disk.dm-0.in‚ 28.7578125 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1352.57568359375 disk.sdb1.out‚0$ memory_percent‚91.96562767028809% disk./.usageGB‚2706.9926376342773 disk.nvme1n1.out‚ 26946.21875) disk./.usagePercent‚75.68864711269086 disk.dm-0.out‚ 127114.625 disk.nvme1n1.in‚ 22.5234375 disk.sdb1.in‚0( proc.memory.availableMB‚ 1316.3515625# proc.memory.rssMB‚ 2983.35546875! disk.nvme0n1.out‚ 100168.40625 disk.md0.out‚ 1320.265625 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0" disk.sdb2.out‚1352.57568359375! disk.sda.out‚1352.57568359375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚6.234375 disk.sda.in‚128.625 disk.md0p1.out‚ 1320.265625 disk.sda2.in‚128.625" disk.sda2.out‚1352.57568359375 disk.md0p1.in‚ 199.5859375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875* proc.memory.percent‚18.208956718444824 cpu‚25.214999853293026 proc.cpu.threads‚17 network.sent‚ 1.7589983e+07¿²«2®- _wandbruntime‚8113  _runtime‚8113^ˆÇ¯Á & lr-AdamW/pg1‚0.0003501109242755663 & lr-AdamW/pg2‚0.0003501109242755663 trainer/global_step‚6799 % _timestamp‚1.7885327296913853e+09 _runtime‚8113.227086552 _step‚555˜bÌ°Æ & lr-AdamW/pg1‚0.0003501109242755663 & lr-AdamW/pg2‚0.0003501109242755663 trainer/global_step‚6799 % _timestamp‚1.7885327296913853e+09 _runtime‚8113.227086552 _step‚555«'™72±- _wandbruntime‚8113  _runtime‚8113]÷Œaû'²õO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141973734 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41201215 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8130403231232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781910884352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126262295 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579907718994432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7782820855808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7781910884352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141973734 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚572125826113024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚572125808110080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172082256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 269499392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553225717687344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314082334 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚ 8113.22865678 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141973732 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141973732 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚6799 % _timestamp‚1.7885327296926923e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314055988 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781910884352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126262295 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579907718994432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7782820855808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7781910884352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141973734 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚572125826113024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚572125808110080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172082256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 269499392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553225717687344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314082334 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚ 8113.22865678 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141973732 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141973732 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚6799 % _timestamp‚1.7885327296926923e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314055988 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579991387528192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885327306571047e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172107087 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8131800913408 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579991387528192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172107087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚6799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561059572079210 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631370285833216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7753928822110 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172133457 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314100979 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7783942329344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚572207427195904 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126279548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314127351 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141993892 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993894 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553304735281888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579990459540480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85073145 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639502110686720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572207445198848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41207053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41206405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85073143 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 314127351 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7783942329344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0™ºJ '¶…N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚572207427195904 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126279548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314127351 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141993892 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993894 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553304735281888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579990459540480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85073145 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639502110686720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572207445198848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41207053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41206405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85073143 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 314127351 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7783942329344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚572207427195904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚639501862172160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631370309773312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561058647064638 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579991387528192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885327306571047e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172107087 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8131800913408 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579991387528192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172107087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚6799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561059572079210 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631370285833216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7753928822110 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172133457 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314100979 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7783942329344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0­ÿ‹Ý2·- _wandbruntime‚8114  _runtime‚8114˜^rîX¸Ò /loss/FRt2/assignment_loss‚3.070018526046507 1loss/FBt2/assignment_loss‚0.03735641063290781 /loss/FBt3/detection_loss‚0.6812793449690081 /loss/FBt4/detection_loss‚2.8730997112062244 0loss/SRqqt2/assignment_loss‚1.16525382931168 /loss/FRt3/detection_loss‚1.4818651947837609 .loss/FBt2/detection_loss‚1.910805540425437 epoch‚5 2loss/SRqqt3/assignment_loss‚0.8409570827656849 0loss/SRqqt1/detection_loss‚2.317563067873319 1loss/SRqqt2/detection_loss‚1.9761648717764262 _runtime‚8114.194809311 1loss/SRqqt1/assignment_loss‚1.472358982016643 1loss/FBt3/assignment_loss‚0.07660334081979084 1loss/SRqqt3/detection_loss‚1.0298634361772037 0loss/FRt4/assignment_loss‚3.0573747868721304 2loss/SRqqt4/assignment_loss‚1.3229913970732943 /loss/FBt1/detection_loss‚2.9679100275039674 _step‚558 .loss/FRt1/assignment_loss‚3.36209604373345 0loss/FRt3/assignment_loss‚0.9644494744447561 1loss/FBt1/assignment_loss‚0.13550992375030546 .loss/FRt1/detection_loss‚3.073985145642207 .loss/FRt4/detection_loss‚2.070008204246943 %loss/total_loss‚40.22068699210256 trainer/global_step‚6799 % _timestamp‚1.7885327306595376e+09 0loss/FBt4/assignment_loss‚0.0857870750074249 /loss/FRt2/detection_loss‚2.4626130368992416 1loss/SRqqt4/detection_loss‚1.7847725381242467Öu¥]¹× 0loss/SRqqt2/assignment_loss‚1.16525382931168 /loss/FRt3/detection_loss‚1.4818651947837609 .loss/FBt2/detection_loss‚1.910805540425437 epoch‚5 2loss/SRqqt3/assignment_loss‚0.8409570827656849 0loss/SRqqt1/detection_loss‚2.317563067873319 1loss/SRqqt2/detection_loss‚1.9761648717764262 _runtime‚8114.194809311 1loss/SRqqt1/assignment_loss‚1.472358982016643 1loss/FBt3/assignment_loss‚0.07660334081979084 1loss/SRqqt3/detection_loss‚1.0298634361772037 0loss/FRt4/assignment_loss‚3.0573747868721304 2loss/SRqqt4/assignment_loss‚1.3229913970732943 /loss/FBt1/detection_loss‚2.9679100275039674 _step‚558 .loss/FRt1/assignment_loss‚3.36209604373345 0loss/FRt3/assignment_loss‚0.9644494744447561 1loss/FBt1/assignment_loss‚0.13550992375030546 .loss/FRt1/detection_loss‚3.073985145642207 .loss/FRt4/detection_loss‚2.070008204246943 %loss/total_loss‚40.22068699210256 trainer/global_step‚6799 % _timestamp‚1.7885327306595376e+09 0loss/FBt4/assignment_loss‚0.0857870750074249 /loss/FRt2/detection_loss‚2.4626130368992416 1loss/SRqqt4/detection_loss‚1.7847725381242467 /loss/FRt2/assignment_loss‚3.070018526046507 1loss/FBt2/assignment_loss‚0.03735641063290781 /loss/FBt3/detection_loss‚0.6812793449690081 /loss/FBt4/detection_loss‚2.8730997112062244®ZŽzk”º‚(:‰ û¯ëÔ­½Îå gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.947' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.48675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T…ži»‚(:Þ û¯ëÔòÒ½é disk.md0.in‚ 199.5859375" disk.nvme0n1.out‚ 100252.984375! disk.sda2.out‚1353.8642578125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚128.625 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 10.84765625$ memory_percent‚91.96281433105469 network.sent‚ 1.7593169e+07 network.recv‚ 2.730567e+06 disk.dm-0.out‚ 127199.375 disk.rbd0.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 1316.8125* proc.memory.percent‚18.208956718444824 cpu‚25.214586318163757( disk./.usagePercent‚75.6938966241003 disk.sda.out‚1353.8642578125 disk.sdb1.in‚0 disk.rbd2.in‚0% disk./.usageGB‚2707.1803855895996! disk.sdb2.out‚1353.8642578125 disk.sda2.in‚128.625# proc.memory.rssMB‚ 2983.35546875 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 25.390625 disk.md0p1.in‚ 199.5859375 disk.sdb.out‚1353.8642578125! disk.nvme1n1.out‚ 26946.390625 disk.md0p1.out‚ 1321.4921875 disk.rbd0.out‚ 2.466796875 disk.md0.out‚ 1321.4921875 disk.dm-0.in‚ 36.23828125 disk.sdb1.out‚0Áh?”¼‚(:‰ Š°ëÔý Ûà gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%VDFk½‚(:à Š°ëԚäâ( proc.memory.availableMB‚ 1316.5390625* proc.memory.percent‚18.208956718444824 network.sent‚ 1.7700163e+07 disk.md0.in‚ 199.59375 disk.md0.out‚ 1323.5625 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚26946.60546875 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0 disk.nvme0n1.in‚ 11.87890625 disk.rbd0.in‚0 disk.sdb1.in‚0$ disk.nvme0n1.out‚100508.96484375 disk.sdb2.out‚1355.994140625 network.recv‚ 2.735649e+06% disk./.usageGB‚2707.4311180114746 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125# memory_percent‚91.9644832611084# proc.memory.rssMB‚ 2983.35546875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.out‚1355.994140625 cpu‚25.214176653128835) disk./.usagePercent‚75.70090720755559 disk.rbd2.out‚ 2.466796875 disk.sda2.out‚1355.994140625 disk.dm-0.in‚ 37.75390625 disk.md0p1.out‚ 1323.5625 disk.dm-0.out‚127446.5703125 disk.sdb.in‚ 70.9609375 disk.sda.out‚1355.994140625 disk.rbd0.out‚ 2.466796875!É ·x¾‚(:í ™°ëÔ´®îà disk.sdb1.in‚0# disk.nvme0n1.out‚100701.3046875$ memory_percent‚91.96288585662842# proc.memory.rssMB‚ 2983.35546875) disk./.usagePercent‚75.70087872919352 disk.rbd0.out‚ 2.466796875 disk.sda1.in‚0 disk.md0.out‚ 1324.78515625! disk.dm-0.out‚127648.10546875 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875) proc.memory.availableMB‚ 1316.80078125 network.sent‚ 1.7706304e+07 disk.sda1.out‚0% disk./.usageGB‚2707.4300994873047 disk.rbd0.in‚0! disk.sda2.out‚1357.2490234375 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚1357.2490234375 proc.cpu.threads‚17 disk.sdb.out‚1357.2490234375 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚26946.80078125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 37.75390625 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1324.78515625 network.recv‚ 2.740081e+06 disk.sda2.in‚ 128.6328125 disk.sda.out‚1357.2490234375* proc.memory.percent‚18.208956718444824 cpu‚25.21376189777694ò?ïv“¿‚(:ˆ ™°ëÔõ¯úà gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.098' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.2745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øCñã“À‚(:ˆ ¨°ëÔž¿¬à gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™uÜrÁ‚(:ç ¨°ëÔí©‚á% disk./.usageGB‚2707.4300994873047 disk.sdb1.in‚0 disk.sda2.out‚1359.654296875 disk.md0.in‚ 199.59375( proc.memory.availableMB‚ 1316.4453125 network.recv‚ 2.744377e+06 disk.sdb2.out‚1359.654296875! disk.nvme1n1.out‚ 26947.453125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 11.87890625 disk.md0.out‚ 1327.12109375 proc.cpu.threads‚17 network.sent‚ 1.7714732e+07 disk.dm-0.in‚ 37.75390625! disk.dm-0.out‚127850.31640625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1359.654296875 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875 disk.md0p1.in‚ 199.59375) disk./.usagePercent‚75.70087872919352 disk.md0p1.out‚ 1327.12109375 disk.rbd0.out‚ 2.466796875 disk.sda1.in‚0$ memory_percent‚91.96505546569824* proc.memory.percent‚18.208956718444824 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1359.654296875 disk.sdb1.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚100902.86328125 disk.sda2.in‚ 128.6328125 cpu‚25.21336901490848# proc.memory.rssMB‚ 2983.35546875 disk.rbd0.in‚06úÍ2Â- _wandbruntime‚8175  _runtime‚8175Âk+NÉÃà ' lr-AdamW/pg1‚0.00034483549308780994 ' lr-AdamW/pg2‚0.00034483549308780994 trainer/global_step‚6849 % _timestamp‚1.7885327914688597e+09 _runtime‚8175.004386721 _step‚559§kÖ”ÎÄÈ % _timestamp‚1.7885327914688597e+09 _runtime‚8175.004386721 _step‚559 ' lr-AdamW/pg1‚0.00034483549308780994 ' lr-AdamW/pg2‚0.00034483549308780994 trainer/global_step‚6849¯ù÷×2Å- _wandbruntime‚8175  _runtime‚8175¹ŽŽ© ÆõO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85664330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584045688685056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576206780404224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚8175.007021101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576206762401280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643980448290304 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segmYÁhQRent.all.allocated‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7808688803940 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85664328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809596778944 _step‚560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7837998309376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142981732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142981732 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173353502 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173327158 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584044760710656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7838908280832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584044760710656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584044760710656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576206762401280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643980448290304 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7808688803940 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85664328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809596778944 _step‚560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7837998309376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142981732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0°÷÷kò”È‚(:‰ ·°ëÔ„•šâ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B<»ÙqÉ‚(:æ ·°ëÔž¶žã disk.sdb1.out‚0 disk.md0.out‚ 1328.57421875$ memory_percent‚91.96505546569824% disk./.usageGB‚2707.4300994873047! disk.sdb2.out‚1361.1708984375 disk.nvme1n1.out‚ 26947.625* proc.memory.percent‚18.208956718444824 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125! disk.sda2.out‚1361.1708984375 network.recv‚ 2.748118e+06 disk.sda.out‚1361.1708984375 disk.sda1.out‚0 disk.md0.in‚ 199.59375( proc.memory.availableMB‚ 1316.4453125 cpu‚25.21296440810271 disk.md0p1.out‚ 1328.57421875 disk.rbd2.out‚ 2.466796875 disk.sdb1.in‚0# disk.nvme0n1.out‚101058.5546875# proc.memory.rssMB‚ 2983.35546875 network.sent‚ 1.7717894e+07 disk.sdb.out‚1361.1708984375 disk.sda.in‚ 128.6328125 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.70087872919352 disk.dm-0.in‚ 37.75390625 disk.dm-0.out‚128006.1796875 disk.nvme1n1.in‚25.875 disk.rbd0.out‚ 2.466796875lG½Z2Ê- _wandbruntime‚8176  _runtime‚8176V¡äp'ËÿM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173378359 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143001892 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127173996 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316380253 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7839119769600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6849 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565066329061518 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316380253 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584128429244416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635881594995712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85676393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576288381487104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584128429244416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576288399490048 $ _timestamp‚1.788532792713929e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7840029754368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8190040972288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚8176.250633214 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143001894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41498223 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚565065404046946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810714052430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644071858445312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635881571055616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143001894 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173351989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85676395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173351989 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143001892 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557255615009088 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127174618 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316353881 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316353881 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557255597969728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7839119769600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173378359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41497603 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7840029754368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584127501256704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8190263449600Œ;oAÌ„N F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6849 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565066329061518 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316380253 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584128429244416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635881594995712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85676393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576288381487104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584128429244416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576288399490048 $ _timestamp‚1.788532792713929e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7840029754368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8190040972288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚8176.250633214 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143001894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41498223 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚565065404046946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810714052430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644071858445312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635881571055616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143001894 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173351989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85676395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173351989 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143001892 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557255615009088 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127174618 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316353881 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316353881 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557255597969728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7839119769600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173378359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41497603 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7840029754368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584127501256704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8190263449600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173378359 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143001892 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127173996 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316380253 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7839119769600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760±—†×@2Í- _wandbruntime‚8176  _runtime‚8176›šú[ÎÕ 0loss/FRt2/assignment_loss‚3.3374471349848642 0loss/FRt3/assignment_loss‚1.3026717529922236 /loss/FRt4/assignment_loss‚2.864154289408428 2loss/SRqqt2/assignment_loss‚1.5736037960276008 _runtime‚8176.252221432 _step‚562 1loss/SRqqt1/assignment_loss‚1.398406448567079 1loss/FBt1/assignment_loss‚0.42665189633784656 1loss/FBt2/assignment_loss‚0.23270388198747802 -loss/FRt2/detection_loss‚2.13369652049409 1loss/SRqqt1/detection_loss‚2.2048270909322634 1loss/SRqqt3/detection_loss‚0.7872265042851165 0loss/FRt1/assignment_loss‚3.1653272379999575 2loss/SRqqt4/assignment_loss‚1.6431372201313144 -loss/FRt1/detection_loss‚3.16800717426383 1loss/SRqqt2/detection_loss‚2.3087386151154834 2loss/FBt3/assignment_loss‚0.004618259440059504 0loss/FBt4/assignment_loss‚0.2772281451487894 /loss/FRt4/detection_loss‚2.4385938073803737 1loss/SRqqt4/detection_loss‚1.9518684032170668 .loss/FBt4/detection_loss‚2.142167870203654 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9379578390329772 /loss/FBt1/detection_loss‚2.7727020723479137 % _timestamp‚1.7885327927166014e+09 /loss/FRt3/detection_loss‚1.4833243256587474 .loss/FBt2/detection_loss‚2.225084504255882 &loss/total_loss‚41.434594556384354 .loss/FBt3/detection_loss‚0.654449766171315 trainer/global_step‚6849ˆÙÆž`ÏÚ &loss/total_loss‚41.434594556384354 .loss/FBt3/detection_loss‚0.654449766171315 trainer/global_step‚6849 0loss/FRt2/assignment_loss‚3.3374471349848642 0loss/FRt3/assignment_loss‚1.3026717529922236 /loss/FRt4/assignment_loss‚2.864154289408428 2loss/SRqqt2/assignment_loss‚1.5736037960276008 _runtime‚8176.252221432 _step‚562 1loss/SRqqt1/assignment_loss‚1.398406448567079 1loss/FBt1/assignment_loss‚0.42665189633784656 1loss/FBt2/assignment_loss‚0.23270388198747802 -loss/FRt2/detection_loss‚2.13369652049409 1loss/SRqqt1/detection_loss‚2.2048270909322634 1loss/SRqqt3/detection_loss‚0.7872265042851165 0loss/FRt1/assignment_loss‚3.1653272379999575 2loss/SRqqt4/assignment_loss‚1.6431372201313144 -loss/FRt1/detection_loss‚3.16800717426383 1loss/SRqqt2/detection_loss‚2.3087386151154834 2loss/FBt3/assignment_loss‚0.004618259440059504 0loss/FBt4/assignment_loss‚0.2772281451487894 /loss/FRt4/detection_loss‚2.4385938073803737 1loss/SRqqt4/detection_loss‚1.9518684032170668 .loss/FBt4/detection_loss‚2.142167870203654 epoch‚5 2loss/SRqqt3/assignment_loss‚0.9379578390329772 /loss/FBt1/detection_loss‚2.7727020723479137 % _timestamp‚1.7885327927166014e+09 /loss/FRt3/detection_loss‚1.4833243256587474 .loss/FBt2/detection_loss‚2.225084504255882²KP|è”Ђ(:‰ Æ°ëÔ—ªÊà gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.469' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.61725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÊòZ`Ñ‚(:Õ Æ°ëÔ®Ó½ä) disk./.usagePercent‚75.70087872919352 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚ 1362.453125 disk.md0.in‚ 199.59375 disk.sda2.in‚ 128.6328125% disk./.usageGB‚2707.4300994873047 disk.sda.out‚ 1362.453125" disk.nvme1n1.out‚ 26947.9453125 disk.md0.out‚ 1329.80078125 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0 disk.nvme0n1.in‚ 11.87890625 disk.nvme0n1.out‚ 101269.3125 proc.cpu.threads‚17 network.recv‚ 2.753003e+06 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0 network.sent‚ 1.7816086e+07 disk.sdb.out‚ 1362.453125 disk.sdb2.out‚ 1362.453125 disk.md0p1.out‚ 1329.80078125 disk.sdb1.in‚0 disk.dm-0.in‚ 37.75390625( proc.memory.availableMB‚ 1399.4140625! proc.memory.rssMB‚ 2900.390625 disk.sda.in‚ 128.6328125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875$ memory_percent‚91.45865440368652* proc.memory.percent‚17.702579498291016 cpu‚25.21253737173242 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚25.875 disk.rbd2.out‚ 2.466796875 disk.dm-0.out‚128217.2578125Ñpõƒ”Ò‚(:‰ Õ°ëÔžŠ¡à gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jËìqÓ‚(:æ Õ°ëÔÔʼnã disk.rbd2.out‚ 2.466796875! disk.sdb2.out‚1363.9833984375! disk.nvme1n1.out‚ 26948.078125! proc.memory.rssMB‚ 2900.390625 disk.md0.in‚ 199.59375$ memory_percent‚91.45703315734863* proc.memory.percent‚17.702579498291016) disk./.usagePercent‚75.68604566231242 disk.rbd2.in‚0 disk.sdb.out‚1363.9833984375( proc.memory.availableMB‚ 1399.6796875 disk.md0p1.out‚ 1331.2734375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1331.2734375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 network.sent‚ 1.7913209e+07% disk./.usageGB‚2706.8995971679688 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚101376.9921875! disk.sda2.out‚1363.9833984375 disk.dm-0.in‚ 37.75390625 disk.dm-0.out‚128325.0703125 disk.rbd0.out‚ 2.466796875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 11.87890625 disk.sdb.in‚ 70.9609375 cpu‚25.212158655380787 disk.sda.out‚1363.9833984375 disk.nvme1n1.in‚25.875 disk.md0p1.in‚ 199.59375 network.recv‚ 2.758629e+06î5”Ô‚(:‰ ä°ëÔ“¡Âà gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A(ænÕ‚(:ã ä°ëÔñ¸øà disk.nvme1n1.out‚ 26948.375 disk.sdb1.in‚0! proc.memory.rssMB‚ 2900.390625 disk.nvme1n1.in‚25.875 disk.sda1.out‚0* proc.memory.percent‚17.702579498291016 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125 disk.sdb1.out‚0$ memory_percent‚91.45698547363281% proc.memory.availableMB‚ 1399.6875) disk./.usagePercent‚75.68445396719156 disk.md0.out‚ 1333.12890625" disk.sdb2.out‚1365.90185546875 disk.rbd0.out‚ 2.466796875 network.sent‚ 1.7917295e+07 network.recv‚ 2.76294e+06 disk.rbd2.in‚0$ disk./.usageGB‚2706.842670440674! disk.sdb.out‚1365.90185546875 disk.rbd0.in‚0 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 37.75390625 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1333.12890625 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd2.out‚ 2.466796875 cpu‚25.21178628839899 disk.nvme0n1.in‚ 11.87890625# disk.nvme0n1.out‚101536.2265625" disk.sda2.out‚1365.90185546875 disk.dm-0.out‚128484.6015625! disk.sda.out‚1365.90185546875nÄ “Ö‚(:ˆ ó°ëÔקÏà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ýüpׂ(:å ó°ëÔ¢£—â disk.sda1.out‚0 disk.dm-0.in‚ 37.75390625" disk.sdb2.out‚1367.44482421875! proc.memory.rssMB‚ 2900.390625) disk./.usagePercent‚75.68680913840134$ disk./.usageGB‚2706.926902770996! disk.sda.out‚1367.44482421875" disk.nvme1n1.out‚ 26948.5703125" disk.sda2.out‚1367.44482421875$ memory_percent‚91.45622253417969 proc.cpu.threads‚17 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1334.6171875 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 network.recv‚ 2.766796e+06 disk.md0.out‚ 1334.6171875 disk.dm-0.out‚ 128674.703125# disk.nvme0n1.out‚101726.1328125 disk.sda2.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625% proc.memory.availableMB‚ 1399.8125 cpu‚25.21139380988212 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875 network.sent‚ 1.7924892e+07 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚17.702579498291016! disk.sdb.out‚1367.44482421875 disk.nvme1n1.in‚25.875X2Ø- _wandbruntime‚8237  _runtime‚8237äÙsƒÇÙÁ _runtime‚8237.055474584 _step‚563 & lr-AdamW/pg1‚0.0003395644516570103 & lr-AdamW/pg2‚0.0003395644516570103 trainer/global_step‚6899 % _timestamp‚1.7885328535199065e+09­ÁÌÚÆ _runtime‚8237.055474584 _step‚563 & lr-AdamW/pg1‚0.0003395644516570103 & lr-AdamW/pg2‚0.0003395644516570103 trainer/global_step‚6899 % _timestamp‚1.7885328535199065e+09³tm‘2Û- _wandbruntime‚8237  _runtime‚82378ê»ü'ÜöO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6899 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7894995705856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143989732 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885328535210414e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318561792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588181802426880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580287734695424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8247656519680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561127460102384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143989734 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143989734 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚580287716692480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8247425640448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7864566059048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588182730401280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7894085734400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41782907 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86267578 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7894995705856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚640302630573568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚568992026161432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588181802426880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143989732 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚580287716692480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86267580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7865474034052 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174572060 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580287734695424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41782287 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561127477141744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ï°M—ÛÝûO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚588182730401280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174572060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚568992951175796 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648550311033344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648550056214016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174598404 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8237.057013741 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318588138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128050487 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128049865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174598404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7894085734400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚640302654513664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588181802426880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580287734695424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8247656519680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561127460102384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143989734 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143989734 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚580287716692480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8247425640448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7864566059048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588182730401280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7894085734400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41782907 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86267578 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7894995705856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚640302630573568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚568992026161432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588181802426880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143989732 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚580287716692480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86267580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 786;“lŒ& 5474034052 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174572060 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580287734695424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41782287 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561127477141744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6899 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7894995705856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143989732 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885328535210414e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318561792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318633155 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318606783 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚580369353781248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640392880218112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569073086043826 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588264542972928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128067659 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561206477696928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚580369353781248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144009892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚640392856278016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865683332326 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8248637694976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚580369335778304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174596891 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174623261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648641493972992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318633155 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8248841297920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588265470960640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚580369335778304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144009894 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚8238.029175089 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615¸ï| 'à„N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128068282 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561206494736288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚588265470960640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144009894 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7896117179392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚648641721516032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86279643 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569072161029254 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144009892 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174623261 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7896117179392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788532854493241e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318606783 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318633155 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318606783 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚580369353781248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640392880218112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569073086043826 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588264542972928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128067659 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561206477696928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚580369353781248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144009892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚640392856278016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865683332326 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8248637694976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚580369335778304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174596891 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174623261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648641493972992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318633155 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8248841297920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588265470960640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚580369335778304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144009894 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚8238.029175089 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7866591307538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768µ{W&è2á- _wandbruntime‚8238  _runtime‚8238 Iùƒ\âÖ _runtime‚8238.030638816 2loss/SRqqt3/assignment_loss‚0.8960883682144873 0loss/FRt4/assignment_loss‚3.1044327398905387 2loss/SRqqt2/assignment_loss‚1.2235885151404593 /loss/FBt1/detection_loss‚2.4075575603379145 /loss/FBt3/detection_loss‚0.7233237298333073 % _timestamp‚1.7885328544953725e+09 1loss/SRqqt4/assignment_loss‚1.332695337710902 .loss/FRt1/detection_loss‚3.168469717502594 0loss/SRqqt4/detection_loss‚2.247963400557637 /loss/FBt2/detection_loss‚2.7438346472653476 epoch‚5 /loss/FRt3/assignment_loss‚1.396202919198506 0loss/SRqqt1/detection_loss‚2.627519216049801 /loss/FRt2/assignment_loss‚2.452157348394394 1loss/FBt2/assignment_loss‚0.16149373351204718 1loss/FBt4/assignment_loss‚0.20241287752738077 .loss/FRt2/detection_loss‚2.495354205625374 /loss/FRt3/detection_loss‚1.3154201447518903 .loss/FRt4/detection_loss‚2.236089827134632 1loss/SRqqt3/detection_loss‚0.9394641259393419 &loss/total_loss‚40.881879075833325 0loss/FRt1/assignment_loss‚2.9978421115875244 .loss/FBt4/detection_loss‚2.638697765090249 _step‚566 2loss/FBt3/assignment_loss‚0.026498731225728697 1loss/SRqqt2/detection_loss‚2.1040746412779154 trainer/global_step‚6899 2loss/SRqqt1/assignment_loss‚1.2324953303587707 1loss/FBt1/assignment_loss‚0.20820208170658183_ÍÿaãÛ % _timestamp‚1.7885328544953725e+09 1loss/SRqqt4/assignment_loss‚1.332695337710902 .loss/FRt1/detection_loss‚3.168469717502594 0loss/SRqqt4/detection_loss‚2.247963400557637 /loss/FBt2/detection_loss‚2.7438346472653476 epoch‚5 /loss/FRt3/assignment_loss‚1.396202919198506 0loss/SRqqt1/detection_loss‚2.627519216049801 /loss/FRt2/assignment_loss‚2.452157348394394 1loss/FBt2/assignment_loss‚0.16149373351204718 1loss/FBt4/assignment_loss‚0.20241287752738077 .loss/FRt2/detection_loss‚2.495354205625374 /loss/FRt3/detection_loss‚1.3154201447518903 .loss/FRt4/detection_loss‚2.236089827134632 1loss/SRqqt3/detection_loss‚0.9394641259393419 &loss/total_loss‚40.881879075833325 0loss/FRt1/assignment_loss‚2.9978421115875244 .loss/FBt4/detection_loss‚2.638697765090249 _step‚566 2loss/FBt3/assignment_loss‚0.026498731225728697 1loss/SRqqt2/detection_loss‚2.1040746412779154 trainer/global_step‚6899 2loss/SRqqt1/assignment_loss‚1.2324953303587707 1loss/FBt1/assignment_loss‚0.20820208170658183 _runtime‚8238.030638816 2loss/SRqqt3/assignment_loss‚0.8960883682144873 0loss/FRt4/assignment_loss‚3.1044327398905387 2loss/SRqqt2/assignment_loss‚1.2235885151404593 /loss/FBt1/detection_loss‚2.4075575603379145 /loss/FBt3/detection_loss‚0.7233237298333073¶Voóº’ä‚(:‡ ‚±ëÔÛ¥Òá gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00úÕxå‚(:í ‚±ëÔÞχã disk.sda1.out‚0 disk.rbd2.out‚ 2.466796875 disk.md0.in‚ 199.59375* proc.memory.percent‚17.702579498291016" disk.sdb2.out‚1368.62255859375 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 11.87890625$ disk./.usageGB‚2706.994197845459 disk.dm-0.in‚ 37.75390625! disk.sdb.out‚1368.62255859375 disk.nvme1n1.in‚25.875 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.466796875 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1368.62255859375 disk.md0.out‚ 1335.74609375! proc.memory.rssMB‚ 2900.390625 network.sent‚ 1.7928354e+07! disk.dm-0.out‚128781.55078125( proc.memory.availableMB‚ 1399.6640625 network.recv‚ 2.770492e+06" disk.nvme1n1.out‚ 26948.8203125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚101832.73046875 disk.sda.in‚ 128.6328125! disk.sda.out‚1368.62255859375 disk.md0p1.out‚ 1335.74609375( disk./.usagePercent‚75.6886907368485 disk.sdb2.in‚ 70.9609375$ memory_percent‚91.45712852478027 cpu‚25.2110049347777442äfæ‚(:Û ‘±ëÔ¦ÿƒá disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 37.75390625 disk.dm-0.out‚128982.9453125 disk.sda.in‚ 128.6328125 disk.sda.out‚1369.978515625 disk.rbd0.in‚0 disk.rbd2.out‚ 2.466796875! proc.memory.rssMB‚ 2900.390625* proc.memory.percent‚17.702579498291016) disk./.usagePercent‚75.69394024825793% disk./.usageGB‚2707.1819458007812" disk.nvme1n1.out‚ 26948.9453125 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 11.87890625 disk.nvme0n1.out‚102034 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.sda2.out‚1369.978515625 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1337.0390625 disk.rbd2.in‚0$ memory_percent‚91.45722389221191( proc.memory.availableMB‚ 1399.6484375 cpu‚25.210605762642857 network.recv‚ 2.776444e+06 disk.sdb.out‚1369.978515625 disk.nvme1n1.in‚25.875 disk.sda1.in‚0 disk.md0.out‚ 1337.0390625 network.sent‚ 1.8038119e+07 disk.sdb2.out‚1369.978515625 disk.sdb1.out‚0ËÎ^û”ç‚(:‰ ‘±ëÔ¼ù”á gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýG9ü”è‚(:‰  ±ëÔŽ·¥à gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xë˧vé‚(:ë  ±ëÔ²‚â disk.md0.in‚ 199.59375" disk.sdb2.out‚1371.94677734375! disk.sda.out‚1371.94677734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚102260.51171875 disk.dm-0.in‚ 37.75390625! disk.sdb.out‚1371.94677734375$ memory_percent‚91.45572185516357 network.recv‚ 2.780877e+06 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1338.953125 disk.md0p1.out‚ 1338.953125 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125 proc.cpu.threads‚17) disk./.usagePercent‚75.70094443208131% disk./.usageGB‚2707.4324493408203 disk.sdb1.out‚0 disk.md0p1.in‚ 199.59375! proc.memory.rssMB‚ 2900.390625 disk.sdb1.in‚0" disk.sda2.out‚1371.94677734375 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚26949.25390625) proc.memory.availableMB‚ 1399.89453125 disk.dm-0.out‚ 129209.765625 disk.sdb.in‚ 70.9609375* proc.memory.percent‚17.702579498291016 network.sent‚ 1.8042062e+07 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 cpu‚25.210219150850943®ÿÈ!”ê‚(:‰ ¯±ëÔ‡à£ß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿKûmë‚(:â ¯±ëÔÈ Ìá disk.sdb.out‚1373.3623046875 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.59375 disk.sdb1.out‚0 disk.nvme0n1.in‚ 11.87890625! disk.sda2.out‚1373.3623046875 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sda.out‚1373.3623046875 cpu‚25.209812716592253 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚25.875 disk.md0p1.out‚ 1340.31640625$ memory_percent‚91.45801067352295) proc.memory.availableMB‚ 1399.51953125* proc.memory.percent‚17.702579498291016 disk.sda1.in‚0 disk.dm-0.out‚ 129411.34375! disk.sdb2.out‚1373.3623046875 network.sent‚ 1.8047546e+07 network.recv‚ 2.785017e+06 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 37.75390625 disk.md0p1.in‚ 199.59375% disk./.usageGB‚2707.4333839416504 disk.rbd0.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.out‚ 102461.875 disk.nvme1n1.out‚ 26949.46875! proc.memory.rssMB‚ 2900.390625 proc.cpu.threads‚17) disk./.usagePercent‚75.70097056391167 disk.md0.out‚ 1340.31640625¨HÓ¿2ì- _wandbruntime‚8299  _runtime‚8299i†xâÉíà ' lr-AdamW/pg1‚0.00033429889846030075 ' lr-AdamW/pg2‚0.00033429889846030075 trainer/global_step‚6949 % _timestamp‚1.7885329156991262e+09 _runtime‚8299.234624169 _step‚567§'[èÎîÈ trainer/global_step‚6949 % _timestamp‚1.7885329156991262e+09 _runtime‚8299.234624169 _step‚567 ' lr-AdamW/pg1‚0.00033429889846030075 ' lr-AdamW/pg2‚0.00033429889846030075·Ü?Jõ2ï- _wandbruntime‚8299  _runtime‚8299Ž\yðöO aMDeviceStat@Ÿ´oè'sMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644813915795968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584368670983680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7951083130880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8305789860352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320814694 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592318844143104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7920443314156 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320841040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653119970961408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885329157003295e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42073787 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8299.236039255 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175816962 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144997732 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7951083130880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7950173159424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175816962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8306031225344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584368670983680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144997734 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86870830 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175843306 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128944617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7950173159424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86870828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592318844143104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42073130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584368688986624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175843306 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128943958 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653119705656320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644813939736064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592319772117504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584368688986624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565078339829584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572999708158104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320841040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653119970961408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885329157003295e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42073787 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8299.236039255 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175816962 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144997732 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7951083130880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7950173159424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175816962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8306031225344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584368670983680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144997734 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86870830 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175843306 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128944617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7950173159424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86870828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592318844143104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42073130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚6949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584368688986624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175843306 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128943958 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653119705656320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644813939736064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592319772117504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584368688986624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565078339829584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572999708158104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320886057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42078885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644904141500416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7921560587434 % _timestamp‚1.7885329166687524e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565157357424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145017894 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145017892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145017894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951294619648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584450308072448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653211362869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644904165440512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175868163 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565157374463488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175868163 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653211126938112 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚592402512676864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592401584689152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚569 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145017892 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128962402 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7922468562646 trainer/global_step‚6949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307197429248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8306985437696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86882893 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175841793 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128961778 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320886057 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320859685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552CGI"¼ô…N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175868163 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565157374463488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175868163 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653211126938112 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚592402512676864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592401584689152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚569 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145017892 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128962402 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7922468562646 trainer/global_step‚6949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_þÒ/×Obatch_end/inactive_split_bytes.all.current‚ 235931648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307197429248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8306985437696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86882893 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175841793 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128961778 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320886057 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320859685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚8300.205472457 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚573079843026134 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7951294619648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320886057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42078885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644904141500416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7921560587434 % _timestamp‚1.7885329166687524e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565157357424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145017894 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145017892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145017894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951294619648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584450308072448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653211362869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644904165440512¹Þ¼³õ2õ- _wandbruntime‚8300  _runtime‚8300~Öh]ö× 2loss/SRqqt2/assignment_loss‚1.6904326312636073 1loss/SRqqt4/assignment_loss‚1.619155733121766 1loss/FBt4/assignment_loss‚0.28786607926018143 .loss/FRt2/detection_loss‚2.251776343898382 /loss/FRt4/detection_loss‚2.2795065780188524 /loss/FBt3/detection_loss‚0.5154475411151846 trainer/global_step‚6949 0loss/FRt3/assignment_loss‚1.4256636044558357 /loss/FBt2/detection_loss‚2.2788187861442566 0loss/FRt4/assignment_loss‚3.4324639346085344 2loss/SRqqt3/assignment_loss‚0.9663977786277731 1loss/FBt1/assignment_loss‚0.13991551650688053 1loss/SRqqt3/detection_loss‚0.8867695979980958 &loss/total_loss‚41.736213273331565 % _timestamp‚1.7885329166711566e+09 _runtime‚8300.206479792 0loss/FBt2/assignment_loss‚0.3439166572468821 0loss/SRqqt4/detection_loss‚2.251174579064051 .loss/FBt1/detection_loss‚2.821654236316681 /loss/FBt4/detection_loss‚2.3223591361727034 .loss/FRt1/detection_loss‚2.718171724501778 _step‚570 0loss/FRt2/assignment_loss‚3.3005489349365233 2loss/SRqqt1/assignment_loss‚1.3509304838048086 1loss/FBt3/assignment_loss‚0.03175667936523174 /loss/FRt3/detection_loss‚1.6670215091138494 0loss/SRqqt1/detection_loss‚2.256199598312378 0loss/FRt1/assignment_loss‚2.9666871042812573 1loss/SRqqt2/detection_loss‚1.9315785051960694 epoch‚5ÿrab÷Ü 1loss/FBt4/assignment_loss‚0.28786607926018143 .loss/FRt2/detection_loss‚2.251776343898382 /loss/FRt4/detection_loss‚2.2795065780188524 /loss/FBt3/detection_loss‚0.5154475411151846 trainer/global_step‚6949 0loss/FRt3/assignment_loss‚1.4256636044558357 /loss/FBt2/detection_loss‚2.2788187861442566 0loss/FRt4/assignment_loss‚3.4324639346085344 2loss/SRqqt3/assignment_loss‚0.9663977786277731 1loss/FBt1/assignment_loss‚0.13991551650688053 1loss/SRqqt3/detection_loss‚0.8867695979980958 &loss/total_loss‚41.736213273331565 % _timestamp‚1.7885329166711566e+09 _runtime‚8300.206479792 0loss/FBt2/assignment_loss‚0.3439166572468821 0loss/SRqqt4/detection_loss‚2.251174579064051 .loss/FBt1/detection_loss‚2.821654236316681 /loss/FBt4/detection_loss‚2.3223591361727034 .loss/FRt1/detection_loss‚2.718171724501778 _step‚570 0loss/FRt2/assignment_loss‚3.3005489349365233 2loss/SRqqt1/assignment_loss‚1.3509304838048086 1loss/FBt3/assignment_loss‚0.03175667936523174 /loss/FRt3/detection_loss‚1.6670215091138494 0loss/SRqqt1/detection_loss‚2.256199598312378 0loss/FRt1/assignment_loss‚2.9666871042812573 1loss/SRqqt2/detection_loss‚1.9315785051960694 epoch‚5 2loss/SRqqt2/assignment_loss‚1.6904326312636073 1loss/SRqqt4/assignment_loss‚1.619155733121766ºî€”ø‚(:‰ ¾±ëÔë£Ãà gpu.0.gpu‚25.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÆèîgù‚(:Ü ¾±ëÔòÆ×â network.recv‚ 2.788093e+06% disk./.usageGB‚2707.4333839416504 disk.sdb.out‚1374.8232421875 disk.sda.out‚1374.8232421875" disk.nvme0n1.out‚ 102703.359375 network.sent‚ 1.8052237e+07 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0' proc.memory.availableMB‚ 1399.828125 cpu‚25.209428155177132 disk.dm-0.in‚ 37.75390625 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1341.7109375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 11.87890625$ memory_percent‚91.45612716674805 disk.nvme1n1.out‚26950) disk./.usagePercent‚75.70097056391167! disk.sdb2.out‚1374.8232421875 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0! proc.memory.rssMB‚ 2900.390625* proc.memory.percent‚17.702579498291016 proc.cpu.threads‚17! disk.sda2.out‚1374.8232421875 disk.md0.in‚ 199.59375 disk.md0.out‚ 1341.7109375 disk.dm-0.out‚ 129653.359375 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚25.875 disk.rbd0.in‚0 disk.sda2.in‚ 128.6328125’GR.’ú‚(:‡ ͱëÔ¦óâà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P¤bytû‚(:é ͱëÔǧùâ disk.sda.out‚1376.8037109375 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625 disk.sdb.out‚1376.8037109375! disk.sda2.out‚1376.8037109375# disk.nvme1n1.out‚26950.13671875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125) proc.memory.availableMB‚ 1399.30859375 network.recv‚ 2.793781e+06# disk.nvme0n1.out‚102833.4921875# memory_percent‚91.4592981338501! disk.sdb2.out‚1376.8037109375 disk.sdb1.in‚0! disk.dm-0.out‚129783.62890625 disk.sdb.in‚ 70.9609375* proc.memory.percent‚17.702579498291016) disk./.usagePercent‚75.70097056391167 disk.dm-0.in‚ 37.75390625 network.sent‚ 1.8162056e+07% disk./.usageGB‚2707.4333839416504 disk.nvme1n1.in‚25.875 disk.md0p1.out‚ 1343.6328125 cpu‚25.208990319518936 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1343.6328125 proc.cpu.threads‚17 disk.sda.in‚ 128.6328125 disk.sda1.out‚0 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2900.390625ã×t“ü‚(:ˆ ܱëÔó¼Ýà gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚149.782' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.4455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾°|fqý‚(:æ ܱëÔÝ¿æá proc.cpu.threads‚17 disk.sdb.out‚1378.2705078125# disk.nvme1n1.out‚26950.42578125 disk.dm-0.out‚129935.6796875( proc.memory.availableMB‚ 1399.5703125) disk./.usagePercent‚75.68716410465224$ disk./.usageGB‚2706.939598083496 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.466796875! disk.sda2.out‚1378.2705078125 disk.md0.out‚ 1345.046875 disk.sdb2.in‚ 70.9609375! proc.memory.rssMB‚ 2900.390625 disk.rbd2.out‚ 2.466796875 disk.sdb.in‚ 70.9609375 cpu‚25.208622859532216 disk.sda1.in‚0$ disk.nvme0n1.out‚102985.25390625 disk.sda2.in‚ 128.6328125 disk.sda.in‚ 128.6328125 disk.sda.out‚1378.2705078125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 11.87890625 disk.md0.in‚ 199.59375$ memory_percent‚91.45770072937012 network.sent‚ 1.8166065e+07! disk.sdb2.out‚1378.2705078125* proc.memory.percent‚17.702579498291016 disk.nvme1n1.in‚25.875 disk.md0p1.out‚ 1345.046875 disk.sda1.out‚0 network.recv‚ 2.798192e+06 disk.rbd2.in‚0 disk.dm-0.in‚ 37.75390625ŸÇì½’þ‚(:‡ ë±ëÔ‰©‹ß gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ö9ùpÿ‚(:å ë±ëÔ¢éïß! proc.memory.rssMB‚ 2900.390625* proc.memory.percent‚17.702579498291016 cpu‚25.20823157583583 network.sent‚ 1.8171394e+07% disk./.usageGB‚2706.2735900878906 disk.md0p1.out‚ 1346.3671875 network.recv‚ 2.802277e+06 disk.sda.out‚1379.642578125 disk.nvme1n1.in‚25.875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 disk.sda2.out‚1379.642578125 disk.dm-0.in‚ 37.75390625 disk.sdb1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb2.out‚1379.642578125 disk.md0.in‚ 199.59375 disk.dm-0.out‚130143.9140625) proc.memory.availableMB‚ 1399.13671875 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1379.642578125 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚26950.62109375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875$ disk.nvme0n1.out‚103193.29296875) disk./.usagePercent‚75.66854224234912 disk.sda.in‚ 128.6328125 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1346.3671875$ memory_percent‚91.46034717559814 ´2€ - _wandbruntime‚8358  _runtime‚8358î´u½Ç Á & lr-AdamW/pg2‚0.0003290399308310756 trainer/global_step‚6999 % _timestamp‚1.7885329753727431e+09 _runtime‚8358.908279886 _step‚571 & lr-AdamW/pg1‚0.0003290399308310756‘ŒZîÌ‚ Æ _runtime‚8358.908279886 _step‚571 & lr-AdamW/pg1‚0.0003290399308310756 & lr-AdamW/pg2‚0.0003290399308310756 trainer/global_step‚6999 % _timestamp‚1.7885329753727431e+09»TÃæN2ƒ - _wandbruntime‚8358  _runtime‚83589Gbû'„ õO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚ 8358.90912752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596456813833728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚657689862009856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87474078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87474080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588449625274880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146005734 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649325224958464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8007170555904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323093942 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8364637051392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177088208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596455885859328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177061864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146005734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42364569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588449643277824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596455885859328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129839287 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8007170555904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177088208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÛF_I… úO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87474080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588449625274880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146005734 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649325224958464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8007170555904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323093942 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8364637051392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177088208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596455885859328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177061864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146005734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42364569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588449643277824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596455885859328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129839287 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8007170555904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177088208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚ 8358.90912752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596456813833728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚657689862009856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87474078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249¼Q-ƒ2† - _wandbruntime‚8359  _runtime‚8359`ôȱ'‡ þM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596539554393088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87486145 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 # _timestamp‚1.78853297633167e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177086695 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8007382044672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42371050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577086600008442 _runtime‚8359.867826694 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323138959 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129857195 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146025894 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569108237151328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177086695 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8008292029440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚588531262363648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42370413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146025892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577085674993870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚6999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129856556 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7977437842542 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596538626405376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚573 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323138959ã ‘& 'ˆ ƒN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177086695 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8007382044672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42371050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577086600008442 _runtime‚8359.867826694 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323138959 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129857195 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146025894 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569108237151328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177086695 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8008292029440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚588531262363648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42370413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146025892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577085674993870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚6999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129856556 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7977437842542 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596538626405376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚573 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323138959 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596539554393088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87486145 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 # _timestamp‚1.78853297633167e+09½ýý 82‰ - _wandbruntime‚8359  _runtime‚8359vÜ<ç]Š × 1loss/SRqqt1/assignment_loss‚2.608502113393375 2loss/SRqqt3/assignment_loss‚1.0442263270308683 2loss/SRqqt4/assignment_loss‚1.5536801209456699 .loss/FBt3/detection_loss‚0.558045036572067 _runtime‚8359.868914149 /loss/FRt1/detection_loss‚2.8766147749764577 .loss/FRt2/detection_loss‚2.494071182818462 _step‚574 .loss/FBt1/detection_loss‚4.378767371177673 2loss/SRqqt2/assignment_loss‚1.6380116103171576 /loss/FRt3/detection_loss‚1.3922017837985088 .loss/FRt4/detection_loss‚2.217148000613237 2loss/FBt3/assignment_loss‚0.030339427682664538 /loss/FBt2/detection_loss‚1.8000233508646488 .loss/FBt4/detection_loss‚2.123030561208725 %loss/total_loss‚41.76499515727801 trainer/global_step‚6999 1loss/SRqqt3/detection_loss‚0.9669860997697809 0loss/SRqqt4/detection_loss‚2.011985102365183 epoch‚5 2loss/FBt1/assignment_loss‚0.002289266526834884 1loss/FBt2/assignment_loss‚0.15562589862344112 1loss/SRqqt1/detection_loss‚2.3980957780565535 /loss/FRt1/assignment_loss‚2.733386222805296 0loss/FRt2/assignment_loss‚2.6409021438779057 0loss/FRt3/assignment_loss‚1.3546672857413857 1loss/FBt4/assignment_loss‚0.19845498135373418 1loss/SRqqt2/detection_loss‚2.0057346937133045 0loss/FRt4/assignment_loss‚2.5822060230450754 % _timestamp‚1.7885329763336413e+09¹ÖÎb‹ Ü /loss/FRt1/assignment_loss‚2.733386222805296 0loss/FRt2/assignment_loss‚2.6409021438779057 0loss/FRt3/assignment_loss‚1.3546672857413857 1loss/FBt4/assignment_loss‚0.19845498135373418 1loss/SRqqt2/detection_loss‚2.0057346937133045 0loss/FRt4/assignment_loss‚2.5822060230450754 % _timestamp‚1.7885329763336413e+09 1loss/SRqqt1/assignment_loss‚2.608502113393375 2loss/SRqqt3/assignment_loss‚1.0442263270308683 2loss/SRqqt4/assignment_loss‚1.5536801209456699 .loss/FBt3/detection_loss‚0.558045036572067 _runtime‚8359.868914149 /loss/FRt1/detection_loss‚2.8766147749764577 .loss/FRt2/detection_loss‚2.494071182818462 _step‚574 .loss/FBt1/detection_loss‚4.378767371177673 2loss/SRqqt2/assignment_loss‚1.6380116103171576 /loss/FRt3/detection_loss‚1.3922017837985088 .loss/FRt4/detection_loss‚2.217148000613237 2loss/FBt3/assignment_loss‚0.030339427682664538 /loss/FBt2/detection_loss‚1.8000233508646488 .loss/FBt4/detection_loss‚2.123030561208725 %loss/total_loss‚41.76499515727801 trainer/global_step‚6999 1loss/SRqqt3/detection_loss‚0.9669860997697809 0loss/SRqqt4/detection_loss‚2.011985102365183 epoch‚5 2loss/FBt1/assignment_loss‚0.002289266526834884 1loss/FBt2/assignment_loss‚0.15562589862344112 1loss/SRqqt1/detection_loss‚2.3980957780565535¾[ñ´ “Œ ‚(:ˆ ú±ëÔìõá gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñy¦g ‚(:Ü ú±ëÔüÞÒá disk.md0.out‚ 1347.8203125* proc.memory.percent‚17.702579498291016 disk.sdb2.out‚ 1381.1484375 disk.sda.in‚ 128.6328125( proc.memory.availableMB‚ 1399.3515625$ disk./.usageGB‚2705.852783203125 disk.dm-0.out‚130231.5234375 disk.nvme1n1.in‚25.875 disk.md0p1.out‚ 1347.8203125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0$ disk.nvme0n1.out‚103280.66015625 disk.sda2.out‚ 1381.1484375 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2900.390625 network.recv‚ 2.806188e+06 disk.dm-0.in‚ 37.75390625 disk.sdb.out‚ 1381.1484375 disk.sdb2.in‚ 70.9609375$ memory_percent‚91.45903587341309 network.sent‚ 1.8189936e+07 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚26950.86328125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.65677630572971 disk.sda.out‚ 1381.1484375 disk.sdb1.out‚0 cpu‚25.207818517785075WuÅ”Ž ‚(:‰ ‰²ëÔçÓµà gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÄÌpm ‚(:â ‰²ëÔвïà disk.md0p1.in‚ 199.59375# disk.nvme0n1.out‚103367.5703125 cpu‚25.207452703992864# disk.nvme1n1.out‚26951.06640625 disk.md0p1.out‚ 1349.66015625! proc.memory.rssMB‚ 2900.390625 disk.sda2.in‚ 128.6328125 network.recv‚ 2.812053e+06 disk.dm-0.in‚ 37.75390625! disk.dm-0.out‚130318.63671875 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1383.052734375 disk.sdb1.in‚0 network.sent‚ 1.8284906e+07' disk./.usagePercent‚75.657547674698# disk./.usageGB‚2705.88037109375 disk.sda.out‚1383.052734375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda2.out‚1383.052734375 disk.md0.out‚ 1349.66015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb.out‚1383.052734375 disk.sda.in‚ 128.6328125 disk.rbd2.in‚0 disk.md0.in‚ 199.59375$ memory_percent‚91.45898818969727* proc.memory.percent‚17.702579498291016 disk.nvme0n1.in‚ 11.87890625 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1399.359375 disk.nvme1n1.in‚25.875 =~’ ‚(:‡ ˜²ëԲʢà gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¡nt‘ ‚(:é ˜²ëÔì¥Âá$ disk.nvme0n1.out‚103368.48828125 network.recv‚ 2.816589e+06 disk.dm-0.in‚ 37.75390625 disk.md0p1.out‚ 1351.09375 disk.rbd0.in‚0 disk.sda1.in‚0! disk.dm-0.out‚130320.19140625" disk.sdb2.out‚1384.54541015625 disk.sda.in‚ 128.6328125 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2900.390625 disk.md0.in‚ 199.59375) proc.memory.availableMB‚ 1399.10546875 cpu‚25.207049617208117 disk.md0p1.in‚ 199.59375 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375! disk.sda.out‚1384.54541015625 disk.rbd2.out‚ 2.466796875 disk.md0.out‚ 1351.09375 network.sent‚ 1.8293354e+07% disk./.usageGB‚2705.8830795288086! disk.sdb.out‚1384.54541015625 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 11.87890625$ memory_percent‚91.46053791046143 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚25.875 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1384.54541015625) disk./.usagePercent‚75.65762340367581! disk.nvme1n1.out‚ 26951.703125 disk.sdb1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.702579498291016ŒbÎ2’ - _wandbruntime‚8411  _runtime‚8411jh:}“ Á _step‚575 & lr-AdamW/pg1‚0.0003237886447303063 & lr-AdamW/pg2‚0.0003237886447303063 trainer/global_step‚|…LJ7049 % _timestamp‚1.7885330278984585e+09 _runtime‚8411.433796836ƒØÌ” Æ & lr-AdamW/pg1‚0.0003237886447303063 & lr-AdamW/pg2‚0.0003237886447303063 trainer/global_step‚7049 % _timestamp‚1.7885330278984585e+09 _runtime‚8411.433796836 _step‚575¿ëMÔ2• - _wandbruntime‚8411  _runtime‚8411üg8æü'– öO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178333110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42656088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130732799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325320498 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚600593855549952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚7049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423251673088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662259761853952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚653836486240768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8032197824372 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147013732 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚592530579566080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88077328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8423035473920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚581012297108356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚600593855549952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600592927575552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147013734 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147013732 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178333110 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚592530597569024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42655471 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚581013222122720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130733418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚662259521714688 _runtime‚8411.434754031 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178306766 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8063257980928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325346844 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8033105799376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006þê„(— ûO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130732799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325320498 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚600593855549952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚7049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423251673088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662259761853952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚653836486240768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8032197824372 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147013732 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚592530579566080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88077328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8423035473920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚581012297108356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚600593855549952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600592927575552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147013734 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147013732 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178333110 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚592530597569024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42655471 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚581013222122720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130733418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚662259521714688 _runtime‚8411.434754031 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178306766 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8063257980928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325346844 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8033105799376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178333110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42656088À¥äœ2˜ - _wandbruntime‚8412  _runtime‚8412+Ä…'™ ÿM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚577 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚592612216654848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8063469469696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚573059116878528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8034223072862 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592612216654848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚573059133917888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325365489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚592612198651904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚592612198651904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033315097650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚8412.385415509 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8064379454464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325365489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42661278 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147033892 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178331597 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130751316 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662351171539968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8424435654656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581092431976178 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88089393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147033894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178357967 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178331597 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600676596109312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147033892 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600675668121600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653926711945216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788533028849453e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42661921 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600676596109312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325391861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚581093356990750 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130750671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147033894 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8424213177344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 trainer/global_step‚7049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88089395 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570c@Lš „N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592612216654848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚573059133917888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325365489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚592612198651904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚592612198651904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033315097650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚8412.385415509 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8064379454464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceSÓ=¸¾tatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325365489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42661278 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147033892 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178331597 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130751316 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662351171539968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8424435654656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581092431976178 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88089393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147033894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178357967 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178331597 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600676596109312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147033892 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600675668121600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653926711945216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788533028849453e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42661921 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600676596109312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325391861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚581093356990750 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130750671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147033894 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8424213177344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 trainer/global_step‚7049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88089395 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚577 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚592612216654848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8063469469696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚573059116878528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8034223072862 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760ÁmÒ}Ö2› - _wandbruntime‚8412  _runtime‚8412Ãl¨bœ Ü /loss/FRt2/detection_loss‚1.9528201558593234 2loss/SRqqt4/assignment_loss‚1.4059763249448116 2loss/FBt3/assignment_loss‚0.036456406880626575 .loss/FBt1/detection_loss‚3.407064124941826 1loss/SRqqt3/detection_loss‚0.8472968270621632 %loss/total_loss‚41.06961338951907 /loss/FRt3/detection_loss‚1.5764765526657387 1loss/SRqqt2/detection_loss‚1.9018773461381595 0loss/FBt3/detection_loss‚0.49407568043814254 trainer/global_step‚7049 2loss/SRqqt3/assignment_loss‚0.8430512334027338 .loss/FBt4/detection_loss‚2.636060726642609 /loss/FRt4/assignment_loss‚3.595390382935019 3loss/FBt1/assignment_loss‚0.0010994468265153046 1loss/FBt2/assignment_loss‚0.21996778437942055 .loss/FRt4/detection_loss‚2.111442380968262 _runtime‚8412.386613644 0loss/FRt1/assignment_loss‚3.1068555489182472 2loss/SRqqt1/assignment_loss‚1.3014671375805682 1loss/FBt4/assignment_loss‚0.24815383877757996 1loss/SRqqt1/detection_loss‚2.5002519036784316 /loss/FBt2/detection_loss‚1.9824246896637812 % _timestamp‚1.7885330288513522e+09 _step‚578 0loss/FRt3/assignment_loss‚1.0840372584633908 2loss/SRqqt2/assignment_loss‚1.4737567886089284 /loss/FRt1/detection_loss‚2.8099337927997112 epoch‚5 0loss/FRt2/assignment_loss‚3.2038782673577466 0loss/SRqqt4/detection_loss‚2.329798789585338ÊSÊg á 1loss/FBt2/assignment_loss‚0.21996778437942055 .loss/FRt4/detection_loss‚2.111442380968262 _runtime‚8412.386613644 0loss/FRt1/assignment_loss‚3.1068555489182472 2loss/SRqqt1/assignment_loss‚1.3014671375805682 1loss/FBt4/assignment_loss‚0.24815383877757996 1loss/SRqqt1/detection_loss‚2.5002519036784316 /loss/FBt2/detection_loss‚1.9824246896637812 % _timestamp‚1.7885330288513522e+09 _step‚578 0loss/FRt3/assignment_loss‚1.0840372584633908 2loss/SRqqt2/assignment_loss‚1.4737567886089284 /loss/FRt1/detection_loss‚2.8099337927997112 epoch‚5 0loss/FRt2/assignment_loss‚3.2038782673577466 0loss/SRqqt4/detection_loss‚2.329798789585338 /loss/FRt2/detection_loss‚1.9528201558593234 2loss/SRqqt4/assignment_loss‚1.4059763249448116 2loss/FBt3/assignment_loss‚0.036456406880626575 .loss/FBt1/detection_loss‚3.407064124941826 1loss/SRqqt3/detection_loss‚0.8472968270621632 %loss/total_loss‚41.06961338951907 /loss/FRt3/detection_loss‚1.5764765526657387 1loss/SRqqt2/detection_loss‚1.9018773461381595 0loss/FBt3/detection_loss‚0.49407568043814254 trainer/global_step‚7049 2loss/SRqqt3/assignment_loss‚0.8430512334027338 .loss/FBt4/detection_loss‚2.636060726642609 /loss/FRt4/assignment_loss‚3.595390382935019 3loss/FBt1/assignment_loss‚0.0010994468265153046Â{¬ß^”ž ‚(:‰ §²ëÔ¹˜’à gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kWø_mŸ ‚(:â §²ëÔ Óà disk.nvme1n1.in‚25.875" disk.nvme1n1.out‚ 26951.9453125 disk.md0p1.out‚ 1352.44140625 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 1385.94921875 disk.rbd0.in‚0 disk.sdb1.in‚0$ disk.nvme0n1.out‚103469.76953125$ memory_percent‚91.46091938018799 proc.cpu.threads‚17 network.recv‚ 2.820184e+06$ disk./.usageGB‚2705.918300628662 disk.md0.in‚ 199.59375 disk.sdb.out‚ 1385.94921875 disk.nvme0n1.in‚ 11.87890625) proc.memory.availableMB‚ 1399.04296875 disk.rbd0.out‚ 2.478515625 disk.sdb1.out‚0 cpu‚25.206634469088797 disk.sda2.in‚ 128.6328125! disk.dm-0.out‚130421.71484375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sda1.out‚0 disk.rbd2.out‚ 2.541015625 disk.md0.out‚ 1352.44140625 disk.dm-0.in‚ 37.75390625 disk.md0p1.in‚ 199.59375 disk.rbd2.in‚0* proc.memory.percent‚17.702579498291016 disk.sda2.out‚ 1385.94921875! proc.memory.rssMB‚ 2900.390625 network.sent‚ 1.829654e+07) disk./.usagePercent‚75.65860820036889 disk.sdb2.out‚ 1385.94921875¼oû.”  ‚(:‰ ¶²ëÔ—ä×à gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Axer¡ ‚(:ç ¶²ëÔÙµîà disk.rbd2.out‚ 2.541015625 disk.sdb2.in‚ 70.9609375 cpu‚25.206171060226225 network.recv‚ 2.825088e+06) disk./.usagePercent‚75.66589652784982$ disk.nvme0n1.out‚103567.50390625 disk.nvme1n1.in‚25.875) proc.memory.availableMB‚ 1398.80078125$ disk./.usageGB‚2706.178966522217 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625! disk.dm-0.out‚130589.29296875 disk.sdb.out‚1387.810546875" disk.nvme1n1.out‚ 27053.9140625 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.541015625* proc.memory.percent‚17.702579498291016 network.sent‚ 1.8403525e+07 disk.md0.in‚ 199.59375 disk.md0p1.out‚ 1354.23828125 disk.sda1.in‚0 disk.sdb2.out‚1387.810546875 disk.sda.out‚1387.810546875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.out‚1387.810546875 disk.md0.out‚ 1354.23828125 disk.dm-0.in‚ 37.75390625 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0$ memory_percent‚91.46239757537842! proc.memory.rssMB‚ 2900.390625׈'½“¢ ‚(:ˆ ŲëÔð£÷Þ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`†oŠf£ ‚(:Û ŲëÔ²¿³à disk.nvme0n1.in‚ 11.87890625 disk.nvme0n1.out‚ 103679.625 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1389.064453125 network.sent‚ 1.8409769e+07) disk./.usagePercent‚75.67300491901308 disk.rbd2.in‚0 disk.dm-0.out‚130775.5703125 disk.sdb2.out‚1389.064453125) proc.memory.availableMB‚ 1399.04296875 disk.rbd0.out‚ 2.541015625 disk.sda1.out‚0 disk.rbd2.out‚ 2.6201171875 disk.rbd0.in‚0 disk.sda.in‚ 128.6328125" disk.nvme1n1.out‚ 27095.9453125 cpu‚25.205690766931724% disk./.usageGB‚2706.4331970214844 disk.md0p1.out‚ 1355.4375 disk.md0.in‚ 199.59375 disk.sdb.out‚1389.064453125$ memory_percent‚91.46091938018799 disk.sdb1.out‚0 disk.sda2.out‚1389.064453125 disk.dm-0.in‚ 37.75390625 network.recv‚ 2.830043e+06 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 disk.md0.out‚ 1355.4375 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚25.875! proc.memory.rssMB‚ 2900.390625* proc.memory.percent‚17.702579498291016 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125 proc.cpu.threads‚17#ïÜÑ“¤ ‚(:ˆ Ô²ëÔûØ…ß gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b¯_6k¥ ‚(:à Ô²ëÔ´‰Åà disk.rbd2.in‚0 disk.md0.in‚ 199.59375! disk.dm-0.out‚130794.61328125$ memory_percent‚91.46249294281006* proc.memory.percent‚17.702579498291016" disk.nvme1n1.out‚ 27096.5703125% disk./.usageGB‚2706.4959831237793 disk.sda2.out‚ 1390.58984375 disk.md0.out‚ 1356.90625 disk.sdb.out‚ 1390.58984375 disk.md0p1.out‚ 1356.90625) disk./.usagePercent‚75.67476044471128 disk.rbd0.out‚ 2.6201171875 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 cpu‚25.205296801668315$ disk.nvme0n1.out‚103698.04296875 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.out‚ 2.6201171875 disk.dm-0.in‚ 37.75390625 disk.md0p1.in‚ 199.59375! proc.memory.rssMB‚ 2900.390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sdb2.out‚ 1390.58984375 disk.nvme1n1.in‚25.875 proc.cpu.threads‚17 network.sent‚ 1.8415921e+07 disk.sda.out‚ 1390.58984375) proc.memory.availableMB‚ 1398.78515625 disk.sda1.in‚0 disk.sda.in‚ 128.6328125 network.recv‚ 2.834379e+06 disk.sdb1.in‚0†gƒ2¦ - _wandbruntime‚8465  _runtime‚8465Ã5Ëȧ  $ _timestamp‚1.788533082003321e+09 _runtime‚8465.538844287 _step‚579 ' lr-AdamW/pg1‚0.00031854613451814555 ' lr-AdamW/pg2‚0.00031854613451814555 trainer/global_step‚7099Mâ¦Í¨ Ç _runtime‚8465.538844287 _step‚579 ' lr-AdamW/pg1‚0.00031854613451814555 ' lr-AdamW/pg2‚0.00031854613451814555 trainer/global_step‚7099 $ _timestamp‚1.788533082003321e+09ÃcË‹82© - _wandbruntime‚8465  _runtime‚8465£Qåû'ª õO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42946181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚580 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179551668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604729969291776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8088075079480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596611551860224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88680578 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚604730897266176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585019054090664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148021734 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8118435434496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680580 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658347771463168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596611533857280 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8119345405952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8119345405952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148021734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8088983054484 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚658347795403264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576930996050544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481669864448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585019979105028 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596611551860224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148021732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666829441327616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481886063616 _runtime‚8465.539947622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604730897266176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327599746 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179551668 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131626759 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 trainer/global_step‚7099 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42946800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615%ï2׫ úO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚576930979011184 $ _timestamp‚1.788533082004354e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481886063616 _runtime‚8465.539947622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604730897266176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327599746 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179551668 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131626759 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 trainer/global_step‚7099 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42946800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42946181 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚580 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179551668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604729969291776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8088075079480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596611551860224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88680578 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚¼¼¸S)604730897266176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585019054090664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148021734 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8118435434496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680580 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658347771463168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596611533857280 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8119345405952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8119345405952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148021734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8088983054484 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚658347795403264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576930996050544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481669864448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585019979105028 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596611551860224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148021732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666829441327616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179602869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666920843475968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8119556894720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596693170946048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148041892 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42952599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596693152943104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148041894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8120466879488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327644763 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88692645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596693152943104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚581 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8089192352758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327644763 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327618391 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482846308352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604812709837824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131645244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658437997167616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8090100327970 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604813637825536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131644605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚7099 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179576499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577009996605728.}á™ '® …N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8090100327970 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604813637825536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131644605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚7099 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179576499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577009996605728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179602869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666920843475968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8119556894720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596693170946048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148041892 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42952599 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596693152943104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148041894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8120466879488 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327644763 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88692645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596693152943104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚581 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8089192352758 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327644763 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327618391 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482846308352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604812709837824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131645244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658437997167616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Åû9›2¯ - _wandbruntime‚8466  _runtime‚8466õ†T°W° Ñ .loss/FRt4/detection_loss‚2.238086607808677 0loss/SRqqt2/detection_loss‚2.216740155365409 0loss/FRt2/assignment_loss‚3.4464250533811507 1loss/FBt1/assignment_loss‚0.17715305788442492 /loss/FBt3/detection_loss‚0.6135603963841938 %loss/total_loss‚43.15263353469831 epoch‚5 _step‚582 /loss/FRt1/assignment_loss‚3.236385781850134 1loss/FBt4/assignment_loss‚0.08212247682321402 2loss/SRqqt1/assignment_loss‚1.9940571578649373 2loss/SRqqt2/assignment_loss‚1.4489726286108902 /loss/FRt2/detection_loss‚2.3430392380431577 .loss/FBt1/detection_loss‚4.060651279985905 .loss/FBt4/detection_loss‚2.349794692463345 2loss/FBt3/assignment_loss‚0.029368928222555466 -loss/FRt1/detection_loss‚2.99862748171602 0loss/SRqqt4/detection_loss‚2.182741072141763 /loss/FBt2/detection_loss‚2.0546424090862274 _runtime‚8466.512258086 % _timestamp‚1.7885330829769776e+09 1loss/SRqqt3/assignment_loss‚1.017498635182924 2loss/SRqqt4/assignment_loss‚1.6975373205813495 /loss/FBt2/assignment_loss‚0.261607146682212 /loss/FRt3/detection_loss‚1.6279555120691658 0loss/SRqqt1/detection_loss‚2.266590063388531 1loss/SRqqt3/detection_loss‚0.7861092657508219 trainer/global_step‚7099 0loss/FRt3/assignment_loss‚1.1100355362892151 /loss/FRt4/assignment_loss‚2.912931637122081œ¶,Ñ\± Ö _step‚582 /loss/FRt1/assignment_loss‚3.236385781850134 1loss/FBt4/assignment_loss‚0.08212247682321402 2loss/SRqqt1/assignment_loss‚1.9940571578649373 2loss/SRqqt2/assignment_loss‚1.4489726286108902 /loss/FRt2/detection_loss‚2.3430392380431577 .loss/FBt1/detection_loss‚4.060651279985905 .loss/FBt4/detection_loss‚2.349794692463345 2loss/FBt3/assignment_loss‚0.029368928222555466 -loss/FRt1/detection_loss‚2.99862748171602 0loss/SRqqt4/detection_loss‚2.182741072141763 /loss/FBt2/detection_loss‚2.0546424090862274 _runtime‚8466.512258086 % _timestamp‚1.7885330829769776e+09 1loss/SRqqt3/assignment_loss‚1.017498635182924 2loss/SRqqt4/assignment_loss‚1.6975373205813495 /loss/FBt2/assignment_loss‚0.261607146682212 /loss/FRt3/detection_loss‚1.6279555120691658 0loss/SRqqt1/detection_loss‚2.266590063388531 1loss/SRqqt3/detection_loss‚0.7861092657508219 trainer/global_step‚7099 0loss/FRt3/assignment_loss‚1.1100355362892151 /loss/FRt4/assignment_loss‚2.912931637122081 .loss/FRt4/detection_loss‚2.238086607808677 0loss/SRqqt2/detection_loss‚2.216740155365409 0loss/FRt2/assignment_loss‚3.4464250533811507 1loss/FBt1/assignment_loss‚0.17715305788442492 /loss/FBt3/detection_loss‚0.6135603963841938 %loss/total_loss‚43.15263353469831 epoch‚5ƶ”² ‚(:‰ ã²ëÔ»øá gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^<£e³ ‚(:Ú ã²ëÔ×ü©á! disk.sdb2.out‚1392.3701171875 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2900.390625 disk.sdb.out‚1392.3701171875 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 2.68310546875 network.recv‚ 2.838386e+06) disk./.usagePercent‚75.67650754422743 disk.sda.out‚1392.3701171875 disk.md0p1.out‚1358.625 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚25.875 disk.sda1.in‚0$ disk.nvme0n1.out‚103860.70703125! disk.sda2.out‚1392.3701171875 disk.md0.out‚1358.625$ memory_percent‚91.45965576171875# proc.memory.availableMB‚1399.25% disk./.usageGB‚2706.5584678649902! disk.dm-0.out‚131009.33203125 disk.nvme1n1.out‚ 27148.625 disk.sdb1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.702579498291016 network.sent‚ 1.8421303e+07 disk.dm-0.in‚ 37.75390625 disk.sdb1.in‚0 cpu‚25.20492951052204 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.6201171875 disk.nvme0n1.in‚ 11.87890625œíÁ”´ ‚(:‰ ò²ëÔôÝœß gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?» lµ ‚(:á ò²ëÔ«™ýà disk.md0p1.out‚ 1360.24609375% disk./.usageGB‚2706.6832008361816 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 37.75390625$ disk.nvme0n1.out‚104023.49609375 disk.dm-0.out‚131276 disk.sdb.out‚1394.048828125 disk.sda.in‚ 128.6328125 disk.rbd0.in‚0 disk.md0.out‚ 1360.24609375 disk.nvme1n1.in‚25.875 disk.rbd2.out‚ 2.68310546875 disk.nvme0n1.in‚ 11.87890625$ memory_percent‚91.46127700805664* proc.memory.percent‚17.702579498291016) disk./.usagePercent‚75.67999513030674 disk.sdb.in‚ 70.9609375 disk.sda.out‚1394.048828125 disk.sda1.in‚0' proc.memory.availableMB‚ 1398.984375! proc.memory.rssMB‚ 2900.390625 network.sent‚ 1.8528468e+07 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.843587e+06 disk.sda2.out‚1394.048828125 disk.sdb2.out‚1394.048828125# disk.nvme1n1.out‚27252.50390625 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd0.out‚ 2.68310546875 cpu‚25.20460007614766Ž¬ß”¶ ‚(:‰ ³ëÔ•ã¹à gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚153.793' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.44825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰™#Ç· ‚(:ì ³ëÔú“³á disk.rbd2.out‚ 2.74609375$ disk.nvme0n1.out‚104078.28515625 disk.md0.in‚ 199.59375! disk.dm-0.out‚131434.61328125 disk.rbd0.out‚ 2.68310546875 cpu‚25.204266866611086 disk.dm-0.in‚ 37.75390625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1361.55078125* proc.memory.percent‚17.702579498291016 disk.md0.out‚ 1361.55078125 disk.md0p1.in‚ 199.59375 network.sent‚ 1.8534602e+07 disk.rbd2.in‚0 disk.nvme0n1.in‚ 11.87890625! disk.sdb.out‚1395.41845703125 disk.sda.in‚ 128.6328125# memory_percent‚91.4604663848877( proc.memory.availableMB‚ 1399.1171875) disk./.usagePercent‚75.67216358073571 disk.sdb1.in‚0" disk.sda2.out‚1395.41845703125 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1395.41845703125! disk.nvme1n1.out‚ 27356.328125! proc.mŠx(×°emory.rssMB‚ 2900.390625 disk.nvme1n1.in‚25.875 disk.rbd0.in‚0 network.recv‚ 2.84844e+06$ disk./.usageGB‚2706.403106689453! disk.sda.out‚1395.41845703125×2YŽ”¸ ‚(:‰ ³ëÔŒ’àà gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ½Á/n¹ ‚(:ã ³ëÔÑô«â! proc.memory.rssMB‚ 2900.390625 proc.cpu.threads‚17 disk.md0p1.out‚ 1362.8125 disk.sda2.in‚ 128.6328125" disk.nvme1n1.out‚ 27460.3671875 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1396.73779296875 network.recv‚ 2.852756e+06 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 199.59375 disk.rbd2.out‚ 2.74609375! disk.sdb.out‚1396.73779296875* proc.memory.percent‚17.702579498291016% disk./.usageGB‚2706.4030380249023 disk.md0.out‚ 1362.8125 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875 cpu‚25.203931697331242 network.sent‚ 1.8540849e+07) disk./.usagePercent‚75.67216166084613 disk.dm-0.out‚ 131538.765625 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚104078.3984375 disk.sda1.in‚0 disk.dm-0.in‚ 37.75390625" disk.sdb2.out‚1396.73779296875 disk.rbd2.in‚0 disk.rbd0.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625" disk.sda2.out‚1396.73779296875$ memory_percent‚91.46115779876709) proc.memory.availableMB‚ 1399.00390625ò¤²˜2º - _wandbruntime‚8524  _runtime‚8524ŸÝ…ŠÈ»  _runtime‚ 8524.74606253 _step‚583 ' lr-AdamW/pg1‚0.00031331349272586376 ' lr-AdamW/pg2‚0.00031331349272586376 trainer/global_step‚7149 % _timestamp‚1.7885331412105787e+09á @5ͼ Ç trainer/global_step‚7149 % _timestamp‚1.7885331412105787e+09 _runtime‚ 8524.74606253 _step‚583 ' lr-AdamW/pg1‚0.00031331349272586376 ' lr-AdamW/pg2‚0.00031331349272586376Ç eØ‚2½ - _wandbruntime‚8524  _runtime‚8524ç2Éü'¾ öO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149029734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8175432830976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580881858738384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149029732 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚600692506151424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589025811072972 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89283830 trainer/global_step‚7149 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180822914 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚600692506151424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132518548 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚608867011008000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149029734 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚600692488148480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885331412112882e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚608867011008000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8144860309592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚600692488148480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚608867938982400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0.åól(¿ ûO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚608867011008000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8144860309592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚600692488148480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚608867938982400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149029734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8175432830976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580881858738384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149029732 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚600692506151424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589025811072972 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89283830 trainer/global_step‚7149 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180822914 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚600692506151424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132518548 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚608867011008000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149029734 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚600692488148480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885331412112882e+09 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580960876332928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8145069607866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚600774107234304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43240480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚608950679541760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89295895 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8541513496576 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329897665 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149049892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885331422454755e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180847771 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43241089 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600774107234304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8541280533504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589106870955366 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580960893372288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8145977583078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329871293 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚600774125237248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8175644319744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671490819826688 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149049894 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8525.781703796 _step‚585 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329897665 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662949306330112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132536984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589105945940794 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8175644319744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8176554304512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132536373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89295893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671490562923520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329871293 XJDeviceStatsMonitor.on_train_batch_end/ina^$©-ctive_split_bytes.large_pool.peak‚ 184953856M$Oå ' …N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚608950679541760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89295895 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8541513496576 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329897665 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149049892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885331422454755e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180847771 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43241089 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600774107234304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8541280533504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589106870955366 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580960893372288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8145977583078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329871293 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚600774125237248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8175644319744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671490819826688 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149049894 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8525.781703796 _step‚585 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329897665 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662949306330112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132536984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589105945940794 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8175644319744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8176554304512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132536373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89295893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671490562923520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329871293 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149049892 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180821401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180821401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662949282390016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580960876332928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8145069607866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚600774107234304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43240480ÉÞô°Ù2à - _wandbruntime‚8525  _runtime‚8525…©îð\Ä Ö 0loss/FRt3/assignment_loss‚0.9758732165351058 1loss/SRqqt4/assignment_loss‚1.388797585341719 3loss/FBt1/assignment_loss‚0.0033742349284390607 1loss/SRqqt3/detection_loss‚0.7723530529625714 1loss/SRqqt4/detection_loss‚2.2944237436017683 epoch‚5 0loss/FRt1/assignment_loss‚3.4720681607723236 1loss/FBt3/assignment_loss‚0.03286093436951571 /loss/FBt4/detection_loss‚2.8139062779290334 %loss/total_loss‚43.05116890503879 2loss/SRqqt2/assignment_loss‚1.8302338147554926 1loss/FBt4/assignment_loss‚0.00832368835712519 /loss/FRt4/detection_loss‚2.1233433885499835 /loss/FBt2/detection_loss‚2.1090260222554207 0loss/FRt2/assignment_loss‚3.1039022992957723 0loss/FRt4/assignment_loss‚2.8403234630823135 .loss/FRt1/detection_loss‚3.137205934524536 1loss/SRqqt1/detection_loss‚2.3529142372069822 /loss/FBt3/detection_loss‚1.0154914834331406 ,loss/FRt2/detection_loss‚2.4458266963158 /loss/FRt3/detection_loss‚1.3508960119815487 1loss/SRqqt1/assignment_loss‚2.053237265636844 .loss/FBt1/detection_loss‚4.214716215928395 _step‚586 2loss/SRqqt3/assignment_loss‚0.8315636040456593 0loss/FBt2/assignment_loss‚0.1313021706555375 1loss/SRqqt2/detection_loss‚1.7492054025737607 trainer/global_step‚7149 $ _timestamp‚1.788533142247447e+09 _runtime‚8525.782739091ÇB}´aÅ Û 0loss/FRt3/assignment_loss‚0.9758732165351058 1loss/SRqqt4/assignment_loss‚1.388797585341719 3loss/FBt1/assignment_loss‚0.0033742349284390607 1loss/SRqqt3/detection_loss‚0.7723530529625714 1loss/SRqqt4/detection_loss‚2.2944237436017683 epoch‚5 0loss/FRt1/assignment_loss‚3.4720681607723236 1loss/FBt3/assignment_loss‚0.03286093436951571 /loss/FBt4/detection_loss‚2.8139062779290334 %loss/total_loss‚43.05116890503879 2loss/SRqqt2/assignment_loss‚1.8302338147554926 1loss/FBt4/assignment_loss‚0.00832368835712519 /loss/FRt4/detection_loss‚2.1233433885499835 /loss/FBt2/detection_loss‚2.1090260222554207 0loss/FRt2/assignment_loss‚3.1039022992957723 0loss/FRt4/assignment_loss‚2.8403234630823135 .loss/FRt1/detection_loss‚3.137205934524536 1loss/SRqqt1/detection_loss‚2.3529142372069822 /loss/FBt3/detection_loss‚1.0154914834331406 ,loss/FRt2/detection_loss‚2.4458266963158 /loss/FRt3/detection_loss‚1.3508960119815487 1loss/SRqqt1/assignment_loss‚2.053237265636844 .loss/FBt1/detection_loss‚4.214716215928395 _step‚586 2loss/SRqqt3/assignment_loss‚0.8315636040456593 0loss/FBt2/assignment_loss‚0.1313021706555375 1loss/SRqqt2/detection_loss‚1.7492054025737607 trainer/global_step‚7149 $ _timestamp‚1.788533142247447e+09 _runtime‚8525.782739091Ê"Úæ‘Æ ‚(:† Ÿ³ëÔâþÖà gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðù~ _Ç ‚(:Ô Ÿ³ëÔŸÁâ disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 disk.sda1.out‚0 disk.nvme0n1.in‚ 11.87890625$ memory_percent‚91.46006107330322) proc.memory.availableMB‚ 1399.18359375! proc.memory.rssMB‚ 2900.390625 disk.sda.out‚ 1398.640625 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 37.75390625 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.74609375! disk.nvme0n1.out‚ 104078.40625 disk.sda2.out‚ 1398.640625 disk.md0.out‚ 1364.64453125 disk.sdb2.out‚ 1398.640625 disk.nvme1n1.in‚25.875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.702579498291016 cpu‚25.203602535450056 network.recv‚ 2.856302e+06) disk./.usagePercent‚75.68579778323577 proc.cpu.threads‚17 network.sent‚ 1.8559135e+07 disk.dm-0.out‚ 131643.359375 disk.sdb.out‚ 1398.640625! disk.nvme1n1.out‚ 27564.953125 disk.rbd0.out‚ 2.74609375 disk.sda1.in‚0 disk.md0p1.out‚ 1364.64453125% disk./.usageGB‚2706.8907318115234 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.593753 E–’È ‚(:‡ ®³ëÔߦ“à gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÒêkÉ ‚(:à ®³ëÔ¬ê…á disk.md0p1.out‚ 1366.12109375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.46146774291992! proc.memory.rssMB‚ 2900.390625$ disk./.usageGB‚2706.933292388916 disk.sdb2.out‚1400.173828125 disk.rbd0.out‚ 2.74609375" disk.nvme0n1.out‚ 104078.546875 disk.sda2.out‚1400.173828125 disk.dm-0.in‚ 37.75390625 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.8655583e+07) disk./.usagePercent‚75.68698779479264 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.sdb.out‚1400.173828125 disk.sda.out‚1400.173828125# disk.nvme1n1.out‚27755.05078125 disk.md0.out‚ 1366.12109375! disk.dm-0.out‚131833.59765625 disk.rbd0.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 1398.953125 cpu‚25.20328437934727* proc.memory.percent‚17.702579498291016 network.recv‚ 2.86142e+06 disk.nvme1n1.in‚25.875ÖÓöA’Ê ‚(:‡ ½³ëÔ’Ôîà gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ…‚nË ‚(:ã ½³ëÔ­þ´â disk.sda1.out‚0 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1401.71826171875 disk.nvme1n1.in‚25.875( proc.memory.availableMB‚ 1399.3359375 network.sent‚ 1.8662128e+07 disk.md0.out‚ 1367.6171875 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.md0.in‚ 199.59375 network.recv‚ 2.866765e+06 disk.sdb1.out‚0$ memory_percent‚91.45913124084473" disk.sda2.out‚1401.71826171875 disk.dm-0.in‚ 37.75390625 disk.dm-0.out‚ 132037.734375 disk.sda.in‚ 128.6328125! disk.sda.out‚1401.71826171875! proc.memory.rssMB‚ 2900.390625" disk.sdb2.out‚1401.71826171875 disk.rbd2.out‚ 2.74609375) disk./.usagePercent‚75.69037895975093% disk./.usageGB‚2707.0545768737793 disk.md0p1.in‚ 199.59375 disk.nvme0n1.in‚ 11.87890625" disk.nvme0n1.out‚ 104078.546875* proc.memory.percent‚17.702579498291016 disk.md0p1.out‚ 1367.6171875 disk.sda1.in‚0 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.out‚ 27959.1875 cpu‚25.20292103896177¤B=¤’Ì ‚(:‡ ̳ëÔ”ëØà gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½UÒfÍ ‚(:Û ̳ëÔ‰ÑÜá$ memory_percent‚91.46132469177246* proc.memory.percent‚17.702579498291016 cpu‚25.20262960672216) disk./.usagePercent‚75.69387401206748 disk.md0p1.in‚ 199.59375 disk.dm-0.in‚ 37.75390625 disk.rbd2.in‚0# disk.nvme0n1.out‚104078.7265625 disk.sda.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.sdb.out‚1403.341796875 proc.cpu.threads‚17 network.sent‚ 1.866522e+07% disk./.usageGB‚2707.1795768737793 disk.md0p1.out‚ 1369.1875 disk.rbd0.in‚0 disk.md0.out‚ 1369.1875 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚28160.80078125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.out‚1403.341796875 disk.sdb2.out‚1403.341796875 disk.sda.out‚1403.341796875 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125( proc.memory.availableMB‚ 1398.9765625! proc.memory.rssMB‚ 2900.390625 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚ 70.9609375 network.recv‚ 2.870599e+06 disk.sda1.in‚0! disk.dm-0.out‚132239.52734375 disk.sdb2.in‚ 70.9609375=Øeñ2Î - _wandbruntime‚8583  _runtime‚858356fÉÏ Ã ' lr-AdamW/pg1‚0.00030809180982816766 ' lr-AdamW/pg2‚0.00030809180982816766 trainer/global_step‚7199 % _timestamp‚1.7885332001048827e+09 _runtime‚8583.640464931 _step‚587ˆ<—ÎÐ È _runtime‚8583.640464931 _step‚587 ' lr-AdamW/pg1‚0.00030809180982816766 ' lr-AdamW/pg2‚0.00030809180982816766 trainer/global_step‚7199 % _timestamp‚1.7885332001048827e+09Ë¥¨WÒ2Ñ - _wandbruntime‚8583  _runtime‚8583ygü'Ò öO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613004052724224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182067816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133412349 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675969296514048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182041472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43525271 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8230610284544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613004052724224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚593033493069644 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604773442439680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚593032568055280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613004980698624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8200737564700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚584832755504944 _step‚588=Ò3Sx Ó ûO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8230610284544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613004052724224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚593033493069644 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604773442439680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚593032568055280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613004980698624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8200737564700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚584832755504944 _step‚588 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613004052724224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182067816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133412349 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675969296514048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182041472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43525271 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185Ì Ür2Ô - _wandbruntime‚8584  _runtime‚8584¦š=š'Õ ÿM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332150567 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613087721257984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8600098832896 $ _timestamp‚1.788533201297615e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604855079528448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332124195 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚7199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89899143 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8232641729536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150057892 _runtime‚8584.833804416 _step‚589 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43531704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8200946862974 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332124195 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133430225 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667460591552512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613087721257984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150057894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8599882647040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604855061525504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667460567612416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8231731744768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676060450259456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚584911773099488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8232641729536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8231731744768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604855061525504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182092673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 132057661441dx' 'Ö „N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8200946862974 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332124195 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133430225 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667460591552512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613087721257984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150057894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8599882647040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604855061525504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667460567612416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8231731744768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676060450259456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚584911773099488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8232641729536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8231731744768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604855061525504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182092673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332150567 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613087721257984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8600098832896 $ _timestamp‚1.788533201297615e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604855079528448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332124195 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚7199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89899143 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8232641729536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150057892 _runtime‚8584.833804416 _step‚589 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43531704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Pð¡Çã Á & lr-AdamW/pg1‚0.0003028821740159468 & lr-AdamW/pg2‚0.0003028821740159468 trainer/global_step‚7249 % _timestamp‚1.7885332587697413e+09 _runtime‚8642.305286884 _step‚591DÇ&Ìä Æ & lr-AdamW/pg1‚0.0003028821740159468 & lr-AdamW/pg2‚0.0003028821740159468 trainer/global_step‚7249 % _timestamp‚1.7885332587697413e+09 _runtime‚8642.305286884 _step‚591Ïä&õB2å - _wandbruntime‚8642  _runtime‚8642_éîû'æ õO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671881651070464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151045734 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚608854396730880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617142022414848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671881627130368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183312718 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 _runtime‚8642.306431509 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334332106 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151045734 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8286697709568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680538941602816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚7249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚617141094440448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8256614819808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8255706844804 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597039325037588 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90490330 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151045732 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8287607681024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90490328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8287607681024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608854396730880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8657539060224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183286374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617141094440448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134306797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43815827 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 $ _timestamp‚1.788533258770811e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597040250051952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183312718 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚617141094440448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8256614819808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8255706844804 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597039325037588 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90490330 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151045732 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8287607681024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90490328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8287607681024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608854396730880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8657539060224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183286374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617141094440448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134306797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43815827 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 $ _timestamp‚1.788533258770811e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856жÙÙ.2è - _wandbruntime‚8643  _runtime‚8643‰Ité €N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608936033819648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183337575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617223834986496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183337575 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134324047 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885332598287554e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134324698 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151065894 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151065892 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334377097 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334403469 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8288729154560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8287819169792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183311205 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8288729154560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608936033819648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680630314036736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚593 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334377097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚680630577231360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚617224762974208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚617224762974208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608936015816704 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334403469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597119459905410 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183311205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 144011427¼L¿'’84 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43821654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8658461201920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8287819169792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671971876774912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚8643.365069016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚608936015816704ݸ¹ª 'ê …N JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43822303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8257732093294 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚671971852834816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚8643.365069016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚608936015816704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608936033819648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183337575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617223834986496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183337575 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134324047 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885332598287554e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134324698 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151065894 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151065892 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334377097 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334403469 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8288729154560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8287819169792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183311205 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8288729154560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608936033819648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680630314036736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚593 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334377097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚680630577231360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚617224762974208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚617224762974208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608936015816704 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334403469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597119459905410 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183311205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43821654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8658461201920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8287819169792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671971876774912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8316547802624 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 335772549 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 188150272 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8316547802624 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8687442571264 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚599935740529638 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚591649585319520 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335772549 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 trainer/global_step‚7253 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8316547802624 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 335772549 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peakâO%'‚ 1189768192 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 188150272 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8316547802624 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8687442571264 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚599935740529638 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚591649585319520 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335772549 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 trainer/global_step‚7253 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 trainer/global_step‚7253 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 335796839 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚683834253914624 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8285945314194 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 188150272 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚683834065764352 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151920332 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 trainer/global_step‚7253 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 335796839 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚683834253914624 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8285945314194 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 188150272 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚683834065764352 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151920332 B&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16259132325649261 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.26895758509635925 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49065670371055603 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997155487537384 particle/accuracy_1_of_6‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31233730912208557 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4955708682537079 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5564516186714172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892807602882385 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1773069053888321 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25660577416419983 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4496996998786926 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5603711605072021 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5862497091293335 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8742690682411194 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30358338356018066 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6036754846572876 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17130878567695618 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20174555480480194 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4877583086490631 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2594429850578308 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5240919589996338 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893417954444885 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35391679406166077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9788375496864319 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_3_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4087491035461426 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17534850537776947 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996321976184845 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 ,jet/accuracy_3_of_6‚0.057692307978868484 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49733835458755493 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28189653158187866 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2754629850387573 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860032200813293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4284721612930298 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20762434601783752 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5098599791526794 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5752708315849304 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882670044898987 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4474138021469116 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27622395753860474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5469419360160828 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14638301730155945 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030471859499812126 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17721690237522125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0848214253783226 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333134651184 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2888889014720917 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47933417558670044 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46465519070625305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15325669944286346 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 +jet/accuracy_3_of_5‚0.12826421856880188 particle/accuracy_5_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.518701434135437 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5025230050086975 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4651903212070465 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3847462832927704 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14415204524993896 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4977061152458191 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44809743762016296 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5334212779998779 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5586419701576233 +jet/accuracy_1_of_1‚0.49450644850730896 ,jet/accuracy_4_of_4‚0.010371146723628044 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.282509446144104 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5067272782325745 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08608465641736984 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46666663885116577 jet/accuracy_6_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030733108520508 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11613669246435165 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.031191738322377205 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49252644181251526 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0359673872590065 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38035711646080017 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.380466103553772 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5445512533187866 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_6‚0.43589746952056885 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 jet/accuracy_5_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0737394243478775 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.027185821905732155 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 epoch‚5 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.984195351600647 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29550132155418396 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5035939812660217 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46466466784477234 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9405254125595093 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.034790948033332825 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47811445593833923 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4021266996860504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5137487649917603 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01692160591483116 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5252482891082764 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2132146954536438 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.515264630317688 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚8722.402216902 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3819286823272705 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41048264503479004 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5235015749931335 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012409157119691372 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021258501801639795 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.035087719559669495 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14532309770584106 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.48924094438552856 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18806307017803192 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5101951956748962 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22175076603889465 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9556267857551575 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42081132531166077 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013510959222912788 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2567761540412903 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42270052433013916 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20106442272663116 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11747512221336365 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09722550958395004 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9907935857772827 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.043716587126255035 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5146090388298035 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5488505959510803 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2675493657588959 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15475517511367798 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2578721344470978 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898694753646851 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7050294280052185 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5919376611709595 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5000000596046448 particle/accuracy_3_of_7‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 trainer/global_step‚7253 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4357142746448517 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5684523582458496 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887914657592773 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2132146954536438 )jet/accuracy_1_of_5‚0.966538667678833 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5472925901412964 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3347006142139435 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8589254021644592 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4908006191253662 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48825302720069885 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *particle/sensitivity‚0.336450457572937 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5359119176864624 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16049382090568542 (particle/f_score‚0.42671042680740356 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19487014412879944 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0384344607591629 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15586653351783752 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9876367449760437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02552664652466774 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4770726263523102 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942994713783264 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019187165424227715 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.054679401218891144 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3749999403953552 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178926110267639 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3172011971473694 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4978070557117462 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7707977890968323 +jet/accuracy_2_of_4‚0.39800387620925903 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5388888120651245 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1767241209745407 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5165367722511292 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995929598808289 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5521024465560913 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075759291648865 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.013082437217235565 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4768518805503845 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _step‚597 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26361241936683655 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21555054187774658 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9861046075820923 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23095235228538513 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4476497173309326 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647979199886322 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49733835458755493 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5006968379020691 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2083011418581009 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt3FBl %àt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2442782074213028 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18687327206134796 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5275423526763916 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.451742947101593 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48250287771224976 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9902697801589966 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23403048515319824 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555555284023285 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 % _timestamp‚1.7885333388645706e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15649090707302094 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_1_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981242418289185 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2669377028942108 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897208213806152 particle/accuracy_3_of_3‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5803571343421936 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2450396716594696 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8828651905059814 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.45413947105407715 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20086705684661865 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1831378936767578 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.095786914229393 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23180757462978363 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908148646354675 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003590325126424432 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5044872760772705 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 jet/accuracy_1_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47965115308761597 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.550505518913269]£âk€!¨ 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3172011971473694 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4978070557117462 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7707977890968323 +jet/accuracy_2_of_4‚0.39800387620925903 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5388888120651245 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1767241209745407 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5165367722511292 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995929598808289 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5521024465560913 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075759291648865 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.013082437217235565 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4768518805503845 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _step‚597 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26361241936683655 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21555054187774658 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9861046075820923 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23095235228538513 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4476497173309326 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647979199886322 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49733835458755493 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5006968379020691 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2083011418581009 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2442782074213028 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18687327206134796 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5275423526763916 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.451742947101593 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48250287771224976 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9902697801589966 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23403048515319824 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555555284023285 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 % _timestamp‚1.7885333388645706e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15649090707302094 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_1_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981242418289185 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2669377028942108 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897208213806152 particle/accuracy_3_of_3‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5803571343421936 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2450396716594696 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8828651905059814 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.45413947105407715 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20086705684661865 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1831378936767578 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.095786914229393 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23180757462978363 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982104480266571 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908148646354675 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003590325126424432 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5044872760772705 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 jet/accuracy_1_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47965115308761597 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.550505518913269 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3046853244304657 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993895292282104 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 +jet/accuracy_3_of_4‚0.07144182920455933 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9898806810379028 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.7105262875556946 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1702248752117157 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19967885315418243 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4651072323322296 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_6_of_6‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05490558594465256 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5614542961120605 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22222782671451569 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 particle/accuracy_6_of_6‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5486539006233215 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11728976666927338 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.988343358039856 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4459965229034424 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5376898050308228 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22938455641269684 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9027778506278992 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03207547217607498 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_4_of_6‚0.03846153989434242 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_7‚0 *loss/val_total_loss‚40.656342513998325 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5382136702537537 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20075757801532745 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.506460964679718 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16259132325649261 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.26895758509635925 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49065670371055603 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997155487537384 particle/accuracy_1_of_6‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31233730912208557 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4955708682537079 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5564516186714172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892807602882385 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1773069053888321 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25660577416419983 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4496996998786926 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5603711605072021 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5862497091293335 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8742690682411194 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30358338356018066 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6036754846572876 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17130878567695618 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20174555480480194 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4877583086490631 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2594429850578308 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5240919589996338 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893417954444885 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35391679406166077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9788375496864319 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_3_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4087491035461426 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17534850537776947 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996321976184845 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 ,jet/accuracy_3_of_6‚0.057692307978868484 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49733835458755493 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28189653158187866 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2754629850387573 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860032200813293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4284721612930298 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20762434601783752 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5098599791526794 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5752708315849304 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882670044898987 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4474138021469116 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27622395753860474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5469419360160828 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14638301730155945 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030471859499812126 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17721690237522125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0848214253783226 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333134651184 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2888889014720917 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47933417558670044 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46465519070625305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15325669944286346 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 +jet/accuracy_3_of_5‚0.12826421856880188 particle/accuracy_5_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.518701434135437 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5025230050086975 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4651903212070465 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3847462832927704 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14415204524993896 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4977061152458191 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44809743762016296 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5334212779998779 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5586419701576233 +jet/accuracy_1_of_1‚0.49450644850730896 ,jet/accuracy_4_of_4‚0.010371146723628044 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.282509446144104 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5067272782325745 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08608465641736984 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46666663885116577 jet/accuracy_6_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030733108520508 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11613669246435165 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.031191738322377205 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49252644181251526 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0359673872590065 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38035711646080017 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.380466103553772 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5445512533187866 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_6‚0.43589746952056885 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 jet/accuracy_5_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0737394243478775 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.027185821905732155 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 epoch‚5 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.984195351600647 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29550132155418396 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5035939812660217 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46466466784477234 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9405254125595093 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.034790948033332825 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.47811445593833923 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4021266996860504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5137487649917603 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01692160591483116 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5252482891082764 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2132146954536438 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.515264630317688 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚8722.402216902 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3819286823272705 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 þ·å7):"Purity/0FRt1SRqqt*FBt/event_purity‚0.41048264503479004 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5235015749931335 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012409157119691372 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021258501801639795 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.035087719559669495 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14532309770584106 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.48924094438552856 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18806307017803192 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5101951956748962 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22175076603889465 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9556267857551575 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42081132531166077 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013510959222912788 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2567761540412903 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42270052433013916 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20106442272663116 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11747512221336365 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09722550958395004 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9907935857772827 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.043716587126255035 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5146090388298035 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5488505959510803 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2675493657588959 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15475517511367798 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2578721344470978 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898694753646851 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7050294280052185 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5919376611709595 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5000000596046448 particle/accuracy_3_of_7‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 trainer/global_step‚7253 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4357142746448517 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5684523582458496 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887914657592773 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2132146954536438 )jet/accuracy_1_of_5‚0.966538667678833 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5472925901412964 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3347006142139435 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8589254021644592 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4908006191253662 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48825302720069885 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *particle/sensitivity‚0.336450457572937 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5359119176864624 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16049382090568542 (particle/f_score‚0.42671042680740356 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19487014412879944 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0384344607591629 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15586653351783752 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9876367449760437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02552664652466774 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4770726263523102 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942994713783264 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019187165424227715 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.054679401218891144 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3749999403953552 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178926110267639ÕÖ&…”!‚(:‰ â´ëÔš¤œá gpu.0.gpu‚62.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÞÅx‚!‚(:í â´ëÔÌŠÑá network.sent‚ 1.9039915e+07! disk.dm-0.out‚134751.03515625 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875" disk.nvme1n1.out‚ 29919.1796875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 11.87890625 disk.rbd2.in‚0) disk./.usagePercent‚75.70090752753718 disk.md0.in‚ 199.59375 disk.rbd0.in‚0$ disk.nvme0n1.out‚104831.85546875 cpu‚25.235284490026984 proc.cpu.threads‚17 network.recv‚ 2.916024e+06! disk.sdb.out‚1420.14794921875! disk.sda.out‚1420.14794921875 disk.dm-0.in‚ 37.75390625 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1420.14794921875 disk.md0p1.out‚ 1385.42578125 disk.md0.out‚ 1385.42578125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1972.80859375! proc.memory.rssMB‚ 2977.140625 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1420.14794921875 disk.sda1.in‚0# memory_percent‚87.9589319229126% disk./.usageGB‚2707.4311294555664 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.74609375 disk.rbd2.out‚ 2.74609375* proc.memory.percent‚18.171024322509766},9Ó’ƒ!‚(:‡ ñ´ëÔ¦ê¸à gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î,(þg„!‚(:Ü ñ´ëÔƒ÷–â proc.memory.availableMB‚1973 disk.md0p1.out‚ 1386.94140625 disk.dm-0.in‚ 37.75390625! disk.dm-0.out‚135512.86328125 disk.sda.out‚1421.7197265625 network.recv‚ 2.922158e+06 disk.md0.in‚ 199.59375" memory_percent‚87.957763671875 disk.sdb.out‚1421.7197265625) disk./.usagePercent‚75.70087744926714 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda2.out‚1421.7197265625 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚30120.67578125 cpu‚25.234878778426676% disk./.usageGB‚2707.4300537109375 disk.rbd2.in‚0 disk.md0.out‚ 1386.94140625* proc.memory.percent‚18.171024322509766 disk.nvme0n1.in‚ 11.87890625 disk.sdb2.in‚ 70.9609375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 network.sent‚ 1.9153456e+07 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.out‚ 105392.1875 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0! disk.sdb2.out‚1421.7197265625ZÇ]Ø”…!‚(:‰ €µëÔψ³à gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.137' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.53425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¼h†!‚(:Ý €µëÔ‘’¨â disk.md0p1.out‚ 1388.5625 disk.sda.out‚1423.3935546875 disk.rbd2.in‚0 disk.nvme1n1.in‚25.875# memory_percent‚87.9556655883789 network.sent‚ 1.9159593e+07$ disk.nvme0n1.out‚105392.37109375 disk.sda.in‚ 128.6328125% disk./.usageGB‚2707.4300575256348 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375! disk.sdb2.out‚1423.3935546875 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.74609375 cpu‚25.23447692110714) disk./.usagePercent‚75.70087755592766 disk.nvme0n1.in‚ 11.87890625! disk.dm-0.out‚135714.70703125 disk.sdb2.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1388.5625 disk.dm-0.in‚ 37.75390625 disk.sdb.out‚1423.3935546875& proc.memory.availableMB‚ 1973.34375 proc.cpu.threads‚17 network.recv‚ 2.926525e+06 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375! disk.sda2.out‚1423.3935546875" disk.nvme1n1.out‚ 30322.3359375 disk.rbd0.in‚0* proc.memory.percent‚18.171024322509766 disk.md0.in‚ 199.59375ŒËÈ“‡!‚(:ˆ µëÔƒØÁÞ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yû=Œuˆ!‚(:ê µëÔöÔà disk.rbd0.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625$ disk.nvme0n1.out‚105392.37109375 disk.sdb.out‚1424.8837890625" disk.nvme1n1.out‚ 30485.1796875! disk.dm-0.out‚135877.55078125$ memory_percent‚87.95788288116455 disk.sda1.out‚0 disk.md0.in‚ 199.59375 cpu‚25.234115430093556) disk./.usagePercent‚75.70088064908309 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sda2.out‚1424.8837890625 disk.md0.out‚ 1389.98828125 disk.sdb2.in‚ 70.9609375) proc.memory.availableMB‚ 1972.98046875 disk.sdb1.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚ 70.9609375 network.recv‚ 2.930511e+06 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1389.98828125 disk.sda2.in‚ 128.6328125 disk.sda.out‚1424.8837890625 disk.sda1.in‚0! disk.sdb2.out‚1424.8837890625 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 37.75390625 network.sent‚ 1.9165079e+07% disk./.usageGB‚2707.4301681518555Óô2‰!- _wandbruntime‚8783  _runtime‚8783ºÙ˜œÈŠ! ' lr-AdamW/pg1‚0.00029768567096949585 ' lr-AdamW/pg2‚0.00029768567096949585 trainer/global_step‚7299 $ _timestamp‚1.788533399669701e+09 _runtime‚8783.205260181 _step‚598Bk>ðÍ‹!Ç _step‚598 ' lr-AdamW/pg1‚0.00029768567096949585 ' lr-AdamW/pg2‚0.00029768567096949585 trainer/global_step‚7299 $ _timestamp‚1.788533399669701e+09 _runtime‚8783.205260181Ö_ ­+2Œ!- _wandbruntime‚8783  _runtime‚8783íøïpü'!öO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185114839 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44433389 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153028722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679937792181760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338117215 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153028720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624727450894336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595911342546576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604253701039558 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8374572702208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8373662730752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153028722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153028720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616353806166528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44432752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8746891343360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885333996707964e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91695986 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624728378868736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624727450894336 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136128736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688684659585024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185088495 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616353806166528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8373662730752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688684916501504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185088495 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616353788163584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616353788163584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91695984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338143561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604254626053922 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679937768241664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚599 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 œ±ºbŽ!ûO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624728378868736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624727450894336 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136128736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688684659585024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185088495 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616353806166528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8373662730752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688684916501504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185088495 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616353788163584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616353788163584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91695984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338143561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604254626053922 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679937768241664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚599 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185114839 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44433389 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153028722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679937792181760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338117215 QCDeviceStatsMonitor.on_train_batch_s¿¯ÆlŸtart/allocation.large_pool.freed‚ 153028720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624727450894336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595911342546576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604253701039558 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8374572702208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8373662730752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153028722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153028720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616353806166528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44432752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8746891343360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885333996707964e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91695986 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231×N2!- _wandbruntime‚8784  _runtime‚8784®-Üë'!ÿM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616435425252352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚616435407249408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185139696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8748281085440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136146391 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8375694175744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616435425252352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚595990377180480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338188578 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604334760921952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185113326 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136147008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚616435407249408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91708049 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8748046025216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185139696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788533400630845e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚8784.166987853 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624811119428096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185113326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91708051 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338162206 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595990360141120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8344383741472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153048882 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153048880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688776298971648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338188578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44438957 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8375694175744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8374784190976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153048880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8374784190976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624810191440384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¸®[½ '‘!„N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624810191440384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616435425252352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚616435407249408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185139696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8748281085440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136146391 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8375694175744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616435425252352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚595990377180480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338188578 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604334760921952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185113326 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136147008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚616435407249408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91708049 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8748046025216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185139696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788533400630845e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚8784.166987853 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624811119428096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185113326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91708051 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338162206 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595990360141120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8344383741472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153048882 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153048880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688776298971648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338188578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44438957 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8375694175744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8374784190976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153048880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8374784190976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784Øäfçø2’!- _wandbruntime‚8784  _runtime‚8784½âÓÖW“!Ñ 0loss/FRt1/assignment_loss‚2.6406316266340366 1loss/FBt3/assignment_loss‚0.14621990589533782 epoch‚6 _step‚601 1loss/SRqqt4/detection_loss‚2.1017447775456963 /loss/FBt2/detection_loss‚2.7305037608513465 trainer/global_step‚7299 $ _timestamp‚1.788533400633271e+09 _runtime‚ 8784.16853911 .loss/FBt1/detection_loss‚2.691849662197961 %loss/total_loss‚40.18654729723165 1loss/FBt4/assignment_loss‚0.17760226949212665 .loss/FRt1/detection_loss‚2.865240651018479 /loss/FRt3/assignment_loss‚1.042215362839077 1loss/SRqqt2/assignment_loss‚1.480896481517248 1loss/SRqqt3/assignment_loss‚0.925427649268937 2loss/SRqqt4/assignment_loss‚1.4024246350475928 /loss/FRt2/assignment_loss‚2.709059568813869 2loss/SRqqt1/assignment_loss‚1.6416188668459655 1loss/SRqqt1/detection_loss‚2.4604070568084717 .loss/FRt2/detection_loss‚2.538422240648879 0loss/FRt4/assignment_loss‚2.7310459838723236 /loss/FRt4/detection_loss‚2.1280646878031066 /loss/FBt3/detection_loss‚0.6963685640653016 1loss/FBt1/assignment_loss‚0.14203265632318107 1loss/SRqqt2/detection_loss‚1.8621541508814183 1loss/FBt2/assignment_loss‚0.12619282186585898 .loss/FRt3/detection_loss‚1.355133156089679 1loss/SRqqt3/detection_loss‚0.8955673882079245 -loss/FBt4/detection_loss‚2.69572337269783&7‘\”!Ö 1loss/SRqqt3/assignment_loss‚0.925427649268937 2loss/SRqqt4/assignment_loss‚1.4024246350475928 /loss/FRt2/assignment_loss‚2.709059568813869 2loss/SRqqt1/assignment_loss‚1.6416188668459655 1loss/SRqqt1/detection_loss‚2.4604070568084717 .loss/FRt2/detection_loss‚2.538422240648879 0loss/FRt4/assignment_loss‚2.7310459838723236 /loss/FRt4/detection_loss‚2.1280646878031066 /loss/FBt3/detection_loss‚0.6963685640653016 1loss/FBt1/assignment_loss‚0.14203265632318107 1loss/SRqqt2/detection_loss‚1.8621541508814183 1loss/FBt2/assignment_loss‚0.12619282186585898 .loss/FRt3/detection_loss‚1.355133156089679 1loss/SRqqt3/detection_loss‚0.8955673882079245 -loss/FBt4/detection_loss‚2.69572337269783 0loss/FRt1/assignment_loss‚2.6406316266340366 1loss/FBt3/assignment_loss‚0.14621990589533782 epoch‚6 _step‚601 1loss/SRqqt4/detection_loss‚2.1017447775456963 /loss/FBt2/detection_loss‚2.7305037608513465 trainer/global_step‚7299 $ _timestamp‚1.788533400633271e+09 _runtime‚ 8784.16853911 .loss/FBt1/detection_loss‚2.691849662197961 %loss/total_loss‚40.18654729723165 1loss/FBt4/assignment_loss‚0.17760226949212665 .loss/FRt1/detection_loss‚2.865240651018479 /loss/FRt3/assignment_loss‚1.042215362839077 1loss/SRqqt2/assignment_loss‚1.480896481517248ÙùŽO“•!‚(:ˆ žµëÔÍ¡¯Þ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Md$Äq–!‚(:æ žµëÔ‰››à! proc.memory.rssMB‚ 2977.140625" disk.sda2.out‚1426.62646484375 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1426.62646484375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1391.671875 network.recv‚ 2.93449e+06( disk./.usagePercent‚75.6715100716554 disk.md0.in‚ 199.59375 disk.md0.out‚ 1391.671875 disk.dm-0.out‚136196.8203125! disk.sda.out‚1426.62646484375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625$ disk.nvme0n1.out‚105392.48046875 disk.sdb2.in‚ 70.9609375" memory_percent‚87.959885597229 proc.cpu.threads‚17% disk./.usageGB‚2706.3797340393066 disk.sda1.in‚0! disk.sdb.out‚1426.62646484375 disk.nvme1n1.in‚25.875# disk.nvme1n1.out‚30804.33984375 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 37.75390625 cpu‚25.233691333754745 disk.sdb1.out‚0 disk.md0p1.in‚ 199.59375 network.sent‚ 1.9181384e+07 disk.rbd2.out‚ 2.74609375) proc.memory.availableMB‚ 1972.65234375·䦔—!‚(:‰ ­µëÔèåøÞ gpu.0.gpu‚18.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¦(Çn˜!‚(:ã ­µëÔ”ûà" disk.sda2.out‚1428.02880859375 disk.dm-0.out‚ 136253.9375 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.sda.in‚ 128.6328125 disk.md0.out‚ 1393.0078125$ memory_percent‚87.95835971832275 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375 disk.dm-0.in‚ 37.75390625! disk.sda.out‚1428.02880859375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 cpu‚25.23335212424406 disk.nvme0n1.in‚ 11.87890625 disk.sda2.in‚ 128.6328125! disk.nvme1n1.out‚ 30861.453125 disk.rbd0.out‚ 2.74609375 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 network.recv‚ 2.940444e+06$ disk./.usageGB‚2705.849578857422! disk.sdb.out‚1428.02880859375 disk.nvme1n1.in‚25.875 disk.rbd0.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1972.90234375 network.sent‚ 1.9276587e+07) disk./.usagePercent‚75.65668671088271" disk.nvme0n1.out‚ 105392.484375" disk.sdb2.out‚1428.02880859375 disk.md0p1.out‚ 1393.0078125* proc.memory.percent‚18.171024322509766 disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17µð{”™!‚(:‰ ¼µëÔ‹žß gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûí gš!‚(:Ü ¼µëÔå½½à disk.sda.out‚ 1429.64453125 disk.sda1.in‚0& proc.memory.availableMB‚ 1972.90625 cpu‚25.23301018768229 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 1429.64453125! disk.dm-0.out‚136254.85546875! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1394.56640625 disk.sdb.in‚ 70.9609375 disk.sdb.out‚ 1429.64453125# disk.nvme1n1.out‚30862.34765625 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.74609375# disk.nvme0n1.out‚105392.5078125 disk.dm-0.in‚ 37.75390625 disk.sdb1.in‚0 disk.sdb1.out‚0 network.sent‚ 1.9282736e+07) disk./.usagePercent‚75.65510269532018 disk.nvme1n1.in‚25.875 disk.rbd2.in‚0 disk.md0.in‚ 199.59375# disk./.usageGB‚2705.79292678833 disk.rbd2.out‚ 2.74609375 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1394.56640625 proc.cpu.threads‚17 network.recv‚ 2.944882e+06 disk.sdb2.out‚ 1429.64453125 disk.nvme0n1.in‚ 11.87890625$ memory_percent‚87.958335876464843 _CØ›!‚(:‰ ˵ëÔ ÜßÞ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.õ`¼enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó6Qsœ!‚(:è ˵ëÔ¡‹Öß disk.dm-0.in‚ 37.75390625 disk.sda.in‚ 128.6328125# disk.nvme1n1.out‚30863.59765625 disk.md0p1.out‚ 1396.6015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚1431.7451171875 disk.nvme1n1.in‚25.875 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0$ disk.nvme0n1.out‚105477.03515625) proc.memory.availableMB‚ 1972.55078125* proc.memory.percent‚18.171024322509766 disk.rbd2.out‚ 2.74609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1396.6015625! disk.sda2.out‚1431.7451171875! proc.memory.rssMB‚ 2977.140625 cpu‚25.232648615237263 network.sent‚ 1.9287955e+07 disk.md0.in‚ 199.59375! disk.sdb2.out‚1431.7451171875 disk.nvme0n1.in‚ 11.87890625 network.recv‚ 2.948017e+06 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚87.96050548553467 disk.dm-0.out‚136299.3828125 disk.sdb.out‚1431.7451171875 disk.sdb2.in‚ 70.9609375) disk./.usagePercent‚75.65494707760382% disk./.usageGB‚2705.7873611450195‚Ðt 2!- _wandbruntime‚8836  _runtime‚8836åeöÂÉž!à _step‚602 ' lr-AdamW/pg1‚0.00029250338363226094 ' lr-AdamW/pg2‚0.00029250338363226094 trainer/global_step‚7349 % _timestamp‚1.7885334527147233e+09 _runtime‚8836.250246403ÆPÿÛΟ!È ' lr-AdamW/pg1‚0.00029250338363226094 ' lr-AdamW/pg2‚0.00029250338363226094 trainer/global_step‚7349 % _timestamp‚1.7885334527147233e+09 _runtime‚8836.250246403 _step‚602ÚÆF‰2 !- _wandbruntime‚8836  _runtime‚8836+ ´:û'¡!õO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚620434760457728 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154036720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92299234 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430660127232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚620434742454784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599862239313136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8399143723094 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚620434742454784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340396463 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚603 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599862222273776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154036722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44721788 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154036720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8805709671424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44721156 $ _timestamp‚1.788533452715718e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628864492610560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137020390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684449053464064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8429750155776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608261383036230 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚6288654205849607ˆ›Õ(¢!úO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚7349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186333397 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154036722 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚620434742454784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599862239313136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8399143723094 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚620434742454784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340396463 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚603 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599862222273776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154036722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44721788 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154036720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8805709671424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44721156 $ _timestamp‚1.788533452715718e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628864492610560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137020390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684449053464064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8429750155776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608261383036230 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628865420584960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚620434760457728 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154036720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92299234 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430660127232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340441480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608340592889688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92311299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8400260996580 trainer/global_step‚7349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154056880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340441480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684539303108608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620516379543552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154056880 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154056882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885334536689157e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620516379543552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154056882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186358228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340415108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92311301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44727592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44726964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186384598 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693345945973760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137038263 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340415108 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8806889282560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608341517904260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8430871616000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620516361540608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684539279168512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8806666805248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599941256907680‰! öV¥!…N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137038263 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340415108 aODeviceStatsMonitor.on_traÌŽ$Sµ%in_batch_end/inactive_split_bytes.small_pool.allocated‚ 8806889282560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608341517904260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8430871616000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620516361540608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684539279168512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8806666805248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599941256907680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340441480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608340592889688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92311299 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8400260996580 trainer/global_step‚7349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154056880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340441480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684539303108608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620516379543552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154056880 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154056882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885334536689157e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620516379543552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154056882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186358228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340415108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92311301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44727592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44726964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186384598 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693345945973760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8864092785152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187578299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45011184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612267215004174 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633001534326784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8485837580800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155044722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚624515696745984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187604643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688960338686464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8863868197376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633002462301184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45011792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885335100370562e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633002462301184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624515714748928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624515714748928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624515696745984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697824206883840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0€"üj´!ûO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8485837580800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155044722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚624515696745984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187604643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688960338686464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8863868197376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633002462301184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45011792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885335100370562e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633002462301184 VHDeviceStatsMonitor.onìEñ_train_batch_start/inactive_split_bytes.all.current‚ 248527872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624515714748928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624515714748928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624515696745984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697824206883840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8454113003198 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155044722 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137913668 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92902484 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187604643 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8486747552256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137914278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633001534326784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603813119040336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342623019 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8485837580800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚8893.572913691 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697824455411712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚688960362626560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8864092785152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187578299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45011184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612267215004174 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633001534326784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ßô„Y2µ!- _wandbruntime‚8894  _runtime‚8894Þ°'¶!ÿM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697915593246208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342694382 $ _timestamp‚1.788533511175739e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633085202860544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689050564390912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155064880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342694382 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612348274886568 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633084274872832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8455230276476 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8486959041024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155064882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8487869025792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92914549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612347349871996 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633085202860544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624597315831808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187629500 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8487869025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137932098 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342668010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624597333834752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45016926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187603130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342668010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486959041024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8865028855296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624597315831808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603892136634880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8894.712385854 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187629500 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8456138251688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187603130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8865261818368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603892119595520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137931475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486959041024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8865028855296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624597315831808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603892136634880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8894.712385854 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187629500 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8456138251688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187603130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8865261818368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603892119595520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137931475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697915593246208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342694382 $ _timestamp‚1.788533511175739e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633085202860544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689050564390912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155064880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342694382 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612348274886568 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633084274872832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8455230276476 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8486959041024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155064882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8487869025792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92914549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612347349871996 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633085202860544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624597315831808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187629500 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8487869025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137932098 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚608à„È'2¸!- _wandbruntime‚8894  _runtime‚8894Ñ^Å|X¹!Ò 1loss/SRqqt4/detection_loss‚2.1814406209935746 /loss/FRt4/detection_loss‚2.5029156913882806 .loss/FBt4/detection_loss‚2.863478680451711 % _timestamp‚1.7885335111782882e+09 .loss/FRt1/assignment_loss‚2.50702736278375 0loss/FRt2/assignment_loss‚2.8457366057804654 0loss/FRt3/assignment_loss‚0.9571229849063175 1loss/FBt3/assignment_loss‚0.05561255979327304 1loss/FBt4/assignment_loss‚0.18071299636115631 /loss/FBt3/detection_loss‚0.6506412752632853 trainer/global_step‚7399 /loss/FRt4/assignment_loss‚3.017150507161492 2loss/SRqqt2/assignment_loss‚1.4768302597105503 1loss/SRqqt2/detection_loss‚1.9101992815732955 .loss/FBt2/detection_loss‚2.457564449310303 _runtime‚ 8894.71376071 _step‚609 2loss/SRqqt1/assignment_loss‚1.2634812635572061 0loss/FBt2/assignment_loss‚0.1270855210248527 1loss/SRqqt4/assignment_loss‚1.526713355133931 /loss/FRt2/detection_loss‚2.1446383825232522 /loss/FBt1/detection_loss‚2.3508856784213674 %loss/total_loss‚39.11682147055983 epoch‚6 1loss/SRqqt3/assignment_loss‚0.806448922321139 1loss/FBt1/assignment_loss‚0.22486891224949349 .loss/FRt1/detection_loss‚2.799793630838394 .loss/FRt3/detection_loss‚1.307259376200152 1loss/SRqqt1/detection_loss‚2.0584294834862584 1loss/SRqqt3/detection_loss‚0.9007836693263316¾Àü¢]º!× 0loss/FRt3/assignment_loss‚0.9571229849063175 1loss/FBt3/assignment_loss‚0.05561255979327304 1loss/FBt4/assignment_loss‚0.18071299636115631 /loss/FBt3/detection_loss‚0.6506412752632853 trainer/global_step‚7399 /loss/FRt4/assignment_loss‚3.017150507161492 2loss/SRqqt2/assignment_loss‚1.4768302597105503 1loss/SRqqt2/detection_loss‚1.9101992815732955 .loss/FBt2/detection_loss‚2.457564449310303 _runtime‚ 8894.71376071 _step‚609 2loss/SRqqt1/assignment_loss‚1.2634812635572061 0loss/FBt2/assignment_loss‚0.1270855210248527 1loss/SRqqt4/assignment_loss‚1.526713355133931 /loss/FRt2/detection_loss‚2.1446383825232522 /loss/FBt1/detection_loss‚2.3508856784213674 %loss/total_loss‚39.11682147055983 epoch‚6 1loss/SRqqt3/assignment_loss‚0.806448922321139 1loss/FBt1/assignment_loss‚0.22486891224949349 .loss/FRt1/detection_loss‚2.799793630838394 .loss/FRt3/detection_loss‚1.307259376200152 1loss/SRqqt1/detection_loss‚2.0584294834862584 1loss/SRqqt3/detection_loss‚0.9007836693263316 1loss/SRqqt4/detection_loss‚2.1814406209935746 /loss/FRt4/detection_loss‚2.5029156913882806 .loss/FBt4/detection_loss‚2.863478680451711 % _timestamp‚1.7885335111782882e+09 .loss/FRt1/assignment_loss‚2.50702736278375 0loss/FRt2/assignment_loss‚2.8457366057804654áÞÛžµq»!‚(:æ ‡¶ëÔžâ›ã disk.rbd0.in‚0 disk.sdb1.in‚0! disk.sdb.out‚1437.82470703125 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.59375$ memory_percent‚87.96067237854004 disk.rbd2.out‚ 2.74609375 disk.md0.in‚ 199.59375 disk.md0.out‚ 1402.42578125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚25.875( proc.memory.availableMB‚ 1972.5234375% disk./.usageGB‚2706.5561332702637 disk.sda1.out‚0! disk.nvme1n1.out‚ 30864.796875 network.sent‚ 1.941338e+07 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda.out‚1437.82470703125 network.recv‚ 2.965914e+06" disk.nvme0n1.out‚ 106100.421875 disk.sda2.in‚ 128.6328125" disk.sdb2.out‚1437.82470703125 disk.md0p1.out‚ 1402.42578125 cpu‚25.231286822352583 disk.rbd0.out‚ 2.74609375 disk.rbd2.in‚0) disk./.usagePercent‚75.67644226798177 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 11.87890625" disk.sda2.out‚1437.82470703125 disk.dm-0.in‚ 37.75390625 disk.dm-0.out‚ 136965.21875* proc.memory.percent‚18.171024322509766ç¼~“¼!‚(:ˆ ‡¶ëÔ±óïá gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&wŠv“½!‚(:ˆ –¶ëÔàáà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B˜,o¾!‚(:ä –¶ëÔŒ„¶â disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1403.6875! disk.sdb.out‚1439.14404296875 proc.cpu.threads‚17 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1403.6875 disk.rbd0.in‚0 disk.rbd2.in‚0* proc.memory.percent‚18.171024322509766 cpu‚25.23098720234806$ disk./.usageGB‚2706.680908203125 disk.sda1.in‚0" disk.sdb2.out‚1439.14404296875 disk.sda.in‚ 128.6328125$ memory_percent‚87.96069622039795) proc.memory.availableMB‚ 1972.51953125$ disk.nvme0n1.out‚106301.68359375! disk.dm-0.out‚137166.66015625! proc.memory.rssMB‚ 2977.140625 disk.sdb2.in‚ 70.9609375) disk./.usagePercent‚75.67993102732694" disk.nvme1n1.out‚ 30864.9765625 network.sent‚ 1.9521046e+07 network.recv‚ 2.971033e+06" disk.sda2.out‚1439.14404296875 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 37.75390625! disk.sda.out‚1439.14404296875 disk.nvme1n1.in‚25.875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 2.74609375Vh“¿!‚(:ˆ ¥¶ëÔÙÚ›ß gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¤IóÀ!‚(:é ¥¶ëÔÖƒäà) disk./.usagePercent‚75.67210481078251" disk.sda2.out‚1440.95556640625 disk.sdb1.in‚0 disk.rbd2.out‚ 2.74609375# disk.nvme1n1.out‚30865.40234375$ disk.nvme0n1.out‚106405.69140625 cpu‚25.230609676073644 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1440.95556640625$ memory_percent‚91.97587966918945* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.dm-0.in‚ 37.75390625 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚25.875 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.nvme0n1.in‚ 11.87890625' proc.memory.availableMB‚ 1314.671875! proc.memory.rssMB‚ 2977.140625 network.sent‚ 1.9527407e+07 disk.md0.out‚ 1405.42578125 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1405.42578125 disk.rbd2.in‚0 disk.md0.in‚ 199.59375 disk.dm-0.out‚ 137271.09375" disk.sdb2.out‚1440.95556640625 disk.sda1.in‚0 network.recv‚ 2.97569e+06% disk./.usageGB‚27y˜C06.4010047912598 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1440.95556640625 disk.sdb1.out‚0E(gg’Á!‚(:‡ ´¶ëÔýÄ©ß gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝS'tÂ!‚(:é ´¶ëÔÙ¸æà disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda.in‚ 128.6328125 disk.sda.out‚1442.9072265625 disk.rbd0.in‚0 disk.sdb1.in‚0! disk.nvme0n1.out‚ 106564.71875 disk.md0.out‚ 1407.30859375 network.sent‚ 1.9533448e+07 disk.md0p1.in‚ 199.59375! disk.sda2.out‚1442.9072265625 disk.md0.in‚ 199.59375* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375 disk.sda2.in‚ 128.6328125( proc.memory.availableMB‚ 1314.1484375) disk./.usagePercent‚75.67209595795833# disk.nvme1n1.out‚30866.08203125 disk.rbd0.out‚ 2.74609375 disk.nvme0n1.in‚ 53.61328125! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 2.74609375 cpu‚25.230251540500063 network.recv‚ 2.979946e+06 disk.sdb.out‚1442.9072265625! disk.sdb2.out‚1442.9072265625% disk./.usageGB‚2706.4006881713867! disk.dm-0.out‚137430.80078125 disk.nvme1n1.in‚ 31.23046875 disk.md0p1.out‚ 1407.30859375 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚91.97907447814941 disk.dm-0.in‚84.84375ÉwQ2Ã!- _wandbruntime‚8952  _runtime‚8952nÑæÆÄ!À & lr-AdamW/pg1‚0.0002821857728214956 & lr-AdamW/pg2‚0.0002821857728214956 trainer/global_step‚7449 % _timestamp‚1.7885335688984406e+09 _runtime‚ 8952.43394646 _step‚610£ç7°ËÅ!Å & lr-AdamW/pg1‚0.0002821857728214956 & lr-AdamW/pg2‚0.0002821857728214956 trainer/global_step‚7449 % _timestamp‚1.7885335688984406e+09 _runtime‚ 8952.43394646 _step‚610âUîÔŽ2Æ!- _wandbruntime‚8952  _runtime‚8952î¬K ü'Ç!öO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚628596651037184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8541925005824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚628596669040128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607763998767536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637138576043008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702394315955200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚7449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344875921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚637139504017408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693471623908864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188823201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156052720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 % _timestamp‚1.7885335688994231e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616274897000846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚628596651037184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344902267 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8542834977280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8541925005824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156052720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138806140 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8510898233310 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8922668106240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚8952.435004705 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚628596669040128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156052722 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616273971986482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156052722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368¢;á^(È!ûO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8542834977280 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188849545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45299787 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚637139504017408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8922439324160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188823201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93505736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚628596669040128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156052722 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616273971986482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156052722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚628596651037184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8541925005824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚628596669040128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607763998767536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637138576043008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702394315955200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚7449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344875921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚637139504017408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693471623908864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188823201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156052720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 % _timestamp‚1.7885335688994231e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616274897000846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚628596651037184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344902267 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8542834977280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8541925005824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156052720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138806140 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8510898233310 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8922668106240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚8952.435004705 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607842999322720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156072882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305503 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8923834259968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚8953.490650835 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628678288125952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156072880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628678270123008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138823934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚637222244576768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628678270123008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344947284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 trainer/global_step‚7449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8543046466048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693561849613312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188874402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628678288125952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93517799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344947284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8543046466048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188848032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138823302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344920912 $ _timestamp‚1.788533569954519e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8543956450816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93517801 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616355031868876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188848032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607843016362080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚702485707813376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637222244576768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306133 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616354106854304ÕÅ<Ç 'Ë!„N FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607842999322720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156072882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305503 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8923834259968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚8953.490650835 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628678288125952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156072880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628678270123008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138823934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚637222244576768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628678270123008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344947284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 trainer/global_step‚7449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8543046466048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693561849613312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188874402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628678288125952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93517799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344947284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8543046466048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188848032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138823302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344920912 $ _timestamp‚1.788533569954519e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8543956450816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93517801 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616355031868876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188848032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607843016362080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚702485707813376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637222244576768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306133 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616354106854304äüÐ’ø2Ì!- _wandbruntime‚8953  _runtime‚8953ÝÔÇý[Í!Õ 1loss/FBt1/assignment_loss‚0.06182009766593767 0loss/FBt2/assignment_loss‚0.1529688274097629 0loss/FBt3/assignment_loss‚0.0956140879129242 .loss/FBt1/detection_loss‚3.149665525981358 /loss/FBt3/detection_loss‚0.4386320348112629 $loss/total_loss‚41.8542085270633 _step‚613 % _timestamp‚1.7885335699569395e+09 /loss/FRt4/assignment_loss‚3.371675738921532 /loss/FRt2/detection_loss‚2.0393154278366743 /loss/FRt3/detection_loss‚1.4718310940013806 /loss/FRt4/detection_loss‚2.2265204458664627 0loss/FRt3/assignment_loss‚1.1208255685551065 1loss/SRqqt2/detection_loss‚2.2242493880422494 0loss/SRqqt4/detection_loss‚2.189162121216456 epoch‚6 trainer/global_step‚7449 /loss/FRt1/detection_loss‚2.8559849794421877 .loss/FBt2/detection_loss‚2.115551597193668 /loss/FBt4/detection_loss‚2.4019969453414283 2loss/SRqqt3/assignment_loss‚0.9731469717949177 0loss/FBt4/assignment_loss‚0.5431589221889274 1loss/SRqqt1/detection_loss‚2.5774726569652557 0loss/FRt1/assignment_loss‚3.3169859051704407 2loss/SRqqt1/assignment_loss‚1.3157449896846498 2loss/SRqqt2/assignment_loss‚1.4207899799496915 2loss/SRqqt4/assignment_loss‚1.5679704777896404 1loss/SRqqt3/detection_loss‚0.9009075348237248 _runtime‚8953.492211232 0loss/FRt2/assignment_loss‚3.3222172084976647°žüv`Î!Ú 0loss/FRt1/assignment_loss‚3.3169859051704407 2loss/SRqqt1/assignment_loss‚1.3157449896846498 2loss/SRqqt2/assignment_loss‚1.4207899799496915 2loss/SRqqt4/assignment_loss‚1.5679704777896404 1loss/SRqqt3/detection_loss‚0.9009075348237248 _runtime‚8953.492211232 0loss/FRt2/assignment_loss‚3.3222172084976647 1loss/FBt1/assignment_loss‚0.06182009766593767 0loss/FBt2/assignment_loss‚0.1529688274097629 0loss/FBt3/assignment_loss‚0.0956140879129242 .loss/FBt1/detection_loss‚3.149665525981358 /loss/FBt3/detection_loss‚0.4386320348112629 $loss/total_loss‚41.8542085270633 _step‚613 % _timestamp‚1.7885335699569395e+09 /loss/FRt4/assignment_loss‚3.371675738921532 /loss/FRt2/detection_loss‚2.0393154278366743 /loss/FRt3/detection_loss‚1.4718310940013806 /loss/FRt4/detection_loss‚2.2265204458664627 0loss/FRt3/assignment_loss‚1.1208255685551065 1loss/SRqqt2/detection_loss‚2.2242493880422494 0loss/SRqqt4/detection_loss‚2.189162121216456 epoch‚6 trainer/global_step‚7449 /loss/FRt1/detection_loss‚2.8559849794421877 .loss/FBt2/detection_loss‚2.115551597193668 /loss/FBt4/detection_loss‚2.4019969453414283 2loss/SRqqt3/assignment_loss‚0.9731469717949177 0loss/FBt4/assignment_loss‚0.5431589221889274 1loss/SRqqt1/detection_loss‚2.5774726569652557åúãŸH“Ï!‚(:ˆ öëÔÛ€Âà gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.0705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e0¼ÊqÐ!‚(:æ öëÔ…ñ¦á disk.sdb1.in‚0 disk.nvme0n1.out‚ 106617.25 disk.sda.in‚ 128.6328125! disk.sda.out‚1444.89794921875) disk./.usagePercent‚75.68573208034798 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375! disk.sdb.out‚1444.89794921875 disk.nvme1n1.in‚ 31.23046875( proc.memory.availableMB‚ 1314.3515625 cpu‚25.22994742987673 proc.cpu.threads‚17 network.sent‚ 1.9538885e+07 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 55.73046875" disk.sda2.out‚1444.89794921875$ memory_percent‚91.97783470153809 disk.rbd0.in‚0 disk.dm-0.in‚ 86.9609375! disk.dm-0.out‚137483.61328125 disk.sdb2.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd0.out‚ 2.74609375 disk.sda1.out‚0 disk.rbd2.in‚0" disk.sdb2.out‚1444.89794921875# disk.nvme1n1.out‚30866.36328125$ disk./.usageGB‚2706.888381958008 disk.md0p1.out‚ 1409.234375 disk.sda1.in‚0 disk.md0.out‚ 1409.234375* proc.memory.percent‚18.171024322509766 network.recv‚ 2.983751e+06 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 disk.md0p1.in‚ 199.59375/4ê’Ñ!‚(:‡ Ò¶ëÔçì‘à gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190094447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚9011.555470064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157060722 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157060722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚641275617759232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611714861455376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚641275617759232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94108984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚632677623331328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8598012430848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45589970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706964003715584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚620281653983154 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706963746799104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981070644224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706963746799104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981070644224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190094447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚9011.555470064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157060722 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157060722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚641275617759232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611714861455376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚641275617759232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94108984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚632677623331328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8598012430848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45589970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706964003715584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640çõ 2Ý!- _wandbruntime‚9012  _runtime‚9012O×{é'Þ!ÿM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚632759242417152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45595695 trainer/global_step‚7499 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94121049 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190092934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190092934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347173814 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139716746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9012.792584809 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94121051 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620360863836612 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157080882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8599133891072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚641359286292992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚632759224414208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8566984786692 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8600043875840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157080880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698073158775808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8982230134784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚611793896089280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45595075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚698073134835712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚641359286292992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚641358358305280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139716124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347200186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347200186 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190119304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8567892761904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707055388910592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707055136201728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ùù¤ 'ß!„N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚641359286292992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚641358358305280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139716124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347200186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347200186 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190119304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8567892761904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707055388910592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707055136201728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚632759242417152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45595695 trainer/global_step‚7499 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94121049 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190092934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190092934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347173814 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139716746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9012.792584809 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94121051 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620360863836612 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157080882 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8599133891072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚641359286292992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚632759224414208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8566984786692 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8600043875840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157080880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698073158775808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8982230134784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚611793896089280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45595075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚698073134835712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0èG‹2à!- _wandbruntime‚9012  _runtime‚9012b»: ]á!× 1loss/FBt1/assignment_loss‚0.15976611059829793 1loss/FBt3/assignment_loss‚0.13814578241051473 1loss/FBt4/assignment_loss‚0.18547314422381334 /loss/FBt3/detection_loss‚0.4486837296871774 trainer/global_step‚7499 _step‚617 /loss/FBt2/detection_loss‚1.8302250227757864 % _timestamp‚1.7885336292586014e+09 .loss/FRt1/detection_loss‚3.159493547061394 /loss/FRt3/detection_loss‚1.6007880747440408 1loss/SRqqt1/detection_loss‚2.1152602742458213 0loss/SRqqt3/detection_loss‚0.874607822766491 2loss/SRqqt3/assignment_loss‚1.1108006066368783 2loss/SRqqt4/assignment_loss‚1.3527561853970251 0loss/FRt3/assignment_loss‚1.1048903074420866 1loss/FBt2/assignment_loss‚0.20136269878584706 /loss/FRt2/detection_loss‚2.2582155705705658 /loss/FBt1/detection_loss‚2.3470865809000454 .loss/FBt4/detection_loss‚2.266000419855118 %loss/total_loss‚40.89461893561655 epoch‚6 0loss/FRt2/assignment_loss‚3.3222334132050024 .loss/FRt4/assignment_loss‚3.43979982038339 /loss/FRt4/detection_loss‚2.2290171703789383 1loss/SRqqt2/detection_loss‚2.1832943947778807 _runtime‚9012.794209696 /loss/FRt1/assignment_loss‚3.065611662535832 2loss/SRqqt1/assignment_loss‚1.4777431639617886 1loss/SRqqt4/detection_loss‚2.2834867287066674 2loss/SRqqt2/assignment_loss‚1.7398767035661473°Í|Hbâ!Ü 2loss/SRqqt3/assignment_loss‚1.1108006066368783 2loss/SRqqt4/assignment_loss‚1.3527561853970251 0loss/FRt3/assignment_loss‚1.1048903074420866 1loss/FBt2/assignment_loss‚0.20136269878584706 /loss/FRt2/detection_loss‚2.2582155705705658 /loss/FBt1/detection_loss‚2.3470865809000454 .loss/FBt4/detection_loss‚2.266000419855118 %loss/total_loss‚40.89461893561655 epoch‚6 0loss/FRt2/assignment_loss‚3.3222334132050024 .loss/FRt4/assignment_loss‚3.43979982038339 /loss/FRt4/detection_loss‚2.2290171703789383 1loss/SRqqt2/detection_loss‚2.1832943947778807 _runtime‚9012.794209696 /loss/FRt1/assignment_loss‚3.065611662535832 2loss/SRqqt1/assignment_loss‚1.4777431639617886 1loss/SRqqt4/detection_loss‚2.2834867287066674 2loss/SRqqt2/assignment_loss‚1.7398767035661473 1loss/FBt1/assignment_loss‚0.15976611059829793 1loss/FBt3/assignment_loss‚0.13814578241051473 1loss/FBt4/assignment_loss‚0.18547314422381334 /loss/FBt3/detection_loss‚0.4486837296871774 trainer/global_step‚7499 _step‚617 /loss/FBt2/detection_loss‚1.8302250227757864 % _timestamp‚1.7885336292586014e+09 .loss/FRt1/detection_loss‚3.159493547061394 /loss/FRt3/detection_loss‚1.6007880747440408 1loss/SRqqt1/detection_loss‚2.1152602742458213 0loss/SRqqt3/detection_loss‚0.874607822766491é0‰³’ã!‚(:‡ ÿ¶ëÔáÙ²à gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e= ~ä!‚(:ó ÿ¶ëÔ¢®á disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 98.1953125 disk.rbd0.in‚0 network.sent‚ 1.9664397e+07 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1417.83203125 disk.sda.in‚ 128.6328125 disk.rbd2.out‚ 2.74609375) disk./.usagePercent‚75.73996234797887 disk.sdb.in‚ 70.9609375$ memory_percent‚91.97866916656494 network.recv‚ 3.001899e+06 disk.dm-0.in‚ 459.19921875" disk.sdb2.out‚1453.75146484375 disk.sda1.out‚0 cpu‚25.228558979324415" disk.sda2.out‚1453.75146484375! disk.dm-0.out‚139083.87109375! disk.sdb.out‚1453.75146484375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17% disk./.usageGB‚2708.8279190063477! disk.sda.out‚1453.75146484375 disk.sda1.in‚0) proc.memory.availableMB‚ 1314.21484375 disk.md0.in‚ 199.59375# disk.nvme1n1.out‚31256.48046875 disk.md0p1.out‚ 1417.83203125 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.rbd2.in‚0" disk.nvme0n1.out‚ 107827.390625* proc.memory.percent‚18.171024322509766 disk.nvme0n1.in‚ 361.00390625«˜V8öå!‚(:æ Ž·ëÔåݧà network.sent‚ 1.9774236e+07 disk.rbd0.out‚ 2.74609375 disk.sda1.in‚0 disk.nvme0n1.in‚ 786.12109375 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2977.140625 cpu‚25.228158869505293 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1419.1875 disk.sda1.out‚0" disk.sdb2.out‚1455.15771484375$ disk./.usageGB‚2712.450958251953" disk.sda2.out‚1455.15771484375 disk.dm-0.in‚ 884.31640625! disk.sdb.out‚1455.15771484375! disk.sda.out‚1455.15771484375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.sda.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 network.recv‚ 3.007134e+06) disk./.usagePercent‚75.84126404164573# disk.nvme0n1.out‚108390.1328125 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1419.1875 disk.nvme1n1.in‚ 98.1953125) proc.memory.availableMB‚ 1314.17578125 disk.rbd2.in‚0 disk.dm-0.out‚ 141305.5625w~õ2{ disk.sdb2.in‚ 70.9609375" disk.nvme1n1.out‚ 32939.8046875$ memory_percent‚91.97890758514404 disk.sdb1.in‚0h5Bž”æ!‚(:‰ Ž·ëÔ̳Ðà gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.403' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.60075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>‚¡”ç!‚(:‰ ·ëÔ‡¡Îà gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ooè!‚(:ä ·ëÔþìà disk.rbd0.in‚0 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1457.00830078125 proc.cpu.threads‚17" disk.sda2.out‚1457.00830078125 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 network.recv‚ 3.01156e+06! disk.nvme1n1.out‚ 33466.265625 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.md0.out‚ 1420.96875 disk.dm-0.in‚ 3525.1640625 disk.sda.in‚ 128.6328125 cpu‚25.227750790211783) disk./.usagePercent‚75.90966309436368 disk.md0p1.in‚ 199.59375# disk.nvme0n1.out‚109822.0234375% disk./.usageGB‚2714.8972396850586 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 3426.96875 disk.dm-0.out‚143288.2890625 network.sent‚ 1.9778216e+07 disk.md0p1.out‚ 1420.96875" disk.sdb2.out‚1457.00830078125# memory_percent‚91.9790267944336 disk.nvme1n1.in‚ 98.1953125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375! disk.sda.out‚1457.00830078125& proc.memory.availableMB‚ 1314.15625#]é“é!‚(:ˆ ¬·ëÔò“§à gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë êqê!‚(:æ ¬·ëÔìÃâà network.recv‚ 3.015859e+06 disk.nvme0n1.in‚ 5086.3828125 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚ 1458.44921875! proc.memory.rssMB‚ 2977.140625 network.sent‚ 1.9786723e+07 disk.rbd2.in‚0 disk.sdb1.in‚0) disk./.usagePercent‚75.95952774641485$ disk.nvme0n1.out‚110988.16796875" disk.nvme1n1.out‚ 33777.1953125$ memory_percent‚91.97900295257568) proc.memory.availableMB‚ 1314.16015625 disk.md0.in‚ 199.59375! disk.dm-0.out‚144708.23828125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 2.74609375 disk.dm-0.in‚ 5184.578125 disk.sdb.out‚ 1458.44921875% disk./.usageGB‚2716.6806411743164 disk.sda2.out‚ 1458.44921875 disk.md0.out‚ 1422.36328125 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1422.36328125* proc.memory.percent‚18.171024322509766 cpu‚25.22740268320042 disk.sda2.in‚ 128.6328125 disk.sda1.out‚0 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 98.1953125 proc.cpu.threads‚17 disk.sda.out‚ 1458.44921875½8q\2ë!- _wandbruntime‚9073  _runtime‚9073i7ÓÒÉì!à ' lr-AdamW/pg1‚0.00027193794183407176 ' lr-AdamW/pg2‚0.00027193794183407176 trainer/global_step‚7549 % _timestamp‚1.7885336900303638e+09 _runtime‚9073.566004668 _step‚618žýÎí!È _step‚618 ' lr-AdamW/pg1‚0.00027193794183407176 ' lr-AdamW/pg2‚0.00027193794183407176 trainer/global_step‚7549 % _timestamp‚1.7885336900303638e+09 _runtime‚9073.566004668ê!¡Úƒ2î!- _wandbruntime‚9073  _runtime‚9073ù¥ü'ï!öO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚615665741182576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8654099855872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9039663364608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚615665758221936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140592117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711534108343296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702494194353664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645412659475456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191313005ñ¨6(ð!ûO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚636758559619584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚636758559619584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 % _timestamp‚1.7885336900311606e+09 _step‚619 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94712234 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8622652743526 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚624288410965462 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9039890049536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚9073.566851482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624287485951098 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158068722 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8654099855872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645413587449856 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158068720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191339349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94712236 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8621744768522 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349408071 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚636758577622528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚711533857718272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645412659475456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645412659475456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191313005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚615665741182576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8654099855872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9039663364608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚615665758221936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140592117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711534108343296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702494194353664 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349453088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚711625235274240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702584420058112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158088882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645496328009216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9041052373504 _runtime‚9074.704708821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158088880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158088882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624367620818920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885336911674306e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349453088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349426716 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140609835 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚620 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚636840178705408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8€¶OŠ¨655221316096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191337836 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191364206 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645496328009216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚615744758777120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8622862041800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚645495400021504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8623770017012 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144%ÂU 'ó!…N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚615744758777120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8622862041800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚645495400021504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8623770017012 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚636840196708352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45885536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94724299 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8656131300864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349453088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚711625235274240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702584420058112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158088882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645496328009216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9041052373504 _runtime‚9074.704708821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158088880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158088882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624367620818920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885336911674306e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349453088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349426716 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140609835 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚620 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚636840178705408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8655221316096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191337836 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191364206 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645496328009216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784ì{c2ô!- _wandbruntime‚9074  _runtime‚9074Eî^Zõ!Ô 1loss/FBt3/assignment_loss‚0.03576148096092719 %loss/total_loss‚41.78350906638749 _runtime‚9074.705948516 /loss/FRt3/detection_loss‚1.5956400780585305 1loss/SRqqt3/detection_loss‚0.9399687123671174 1loss/SRqqt4/detection_loss‚1.8768682226538658 0loss/SRqqt3/assignment_loss‚0.87786327669397 1loss/SRqqt4/assignment_loss‚1.119458700530231 .loss/FBt4/detection_loss‚4.219439089298248 trainer/global_step‚7549 /loss/FRt3/assignment_loss‚1.572822950009642 _step‚621 /loss/FRt4/detection_loss‚2.3451086386311344 epoch‚6 % _timestamp‚1.7885336911705341e+09 2loss/SRqqt1/assignment_loss‚1.4583285622494786 1loss/FBt4/assignment_loss‚0.23740065097751994 .loss/FRt2/detection_loss‚2.138353994011038 0loss/SRqqt1/detection_loss‚2.167784151278044 0loss/FRt2/assignment_loss‚3.2996026777452037 0loss/FRt4/assignment_loss‚2.5176990017658327 1loss/FBt2/assignment_loss‚0.11567488582929375 .loss/FRt1/detection_loss‚2.979968622878746 /loss/FBt1/detection_loss‚2.6375027809824263 0loss/FRt1/assignment_loss‚3.2337692512406244 1loss/SRqqt2/assignment_loss‚1.172724592462182 0loss/FBt1/assignment_loss‚0.3336292395809518 1loss/SRqqt2/detection_loss‚2.6673707151412964 /loss/FBt2/detection_loss‚1.9307651277631521 0loss/FBt3/detection_loss‚0.31000366327803464ãU‰&_ö!Ù 0loss/SRqqt1/detection_loss‚2.167784151278044 0loss/FRt2/assignment_loss‚3.2996026777452037 0loss/FRt4/assignment_loss‚2.5176990017658327 1loss/FBt2/assignment_loss‚0.11567488582929375 .loss/FRt1/detection_loss‚2.979968622878746 /loss/FBt1/detection_loss‚2.6375027809824263 0loss/FRt1/assignment_loss‚3.2337692512406244 1loss/SRqqt2/assignment_loss‚1.172724592462182 0loss/FBt1/assignment_loss‚0.3336292395809518 1loss/SRqqt2/detection_loss‚2.6673707151412964 /loss/FBt2/detection_loss‚1.9307651277631521 0loss/FBt3/detection_loss‚0.31000366327803464 1loss/FBt3/assignment_loss‚0.03576148096092719 %loss/total_loss‚41.78350906638749 _runtime‚9074.705948516 /loss/FRt3/detection_loss‚1.5956400780585305 1loss/SRqqt3/detection_loss‚0.9399687123671174 1loss/SRqqt4/detection_loss‚1.8768682226538658 0loss/SRqqt3/assignment_loss‚0.87786327669397 1loss/SRqqt4/assignment_loss‚1.119458700530231 .loss/FBt4/detection_loss‚4.219439089298248 trainer/global_step‚7549 /loss/FRt3/assignment_loss‚1.572822950009642 _step‚621 /loss/FRt4/detection_loss‚2.3451086386311344 epoch‚6 % _timestamp‚1.7885336911705341e+09 2loss/SRqqt1/assignment_loss‚1.4583285622494786 1loss/FBt4/assignment_loss‚0.23740065097751994 .loss/FRt2/detection_loss‚2.138353994011038íâH””÷!‚(:‰ »·ëÔ‰ëŠà gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"íòËvø!‚(:ë »·ëÔÂÆúà disk.sda.in‚ 128.6328125 disk.sdb1.in‚0# disk.nvme0n1.out‚112525.6171875! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.md0p1.out‚ 1423.94140625) disk./.usagePercent‚76.04922872062654! disk.sdb.out‚1460.09521484375 disk.nvme1n1.in‚ 98.1953125 disk.rbd2.in‚0 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0% disk./.usageGB‚2719.8887825012207! disk.sda.out‚1460.09521484375 disk.sda1.out‚0 disk.nvme0n1.in‚ 6781.5625 disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 6879.7578125 cpu‚25.227054289311464 disk.sdb1.out‚0" disk.sda2.out‚1460.09521484375 network.recv‚ 3.01901e+06 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.rbd2.out‚ 2.74609375 disk.md0.in‚ 199.59375 disk.dm-0.out‚147642.2734375$ memory_percent‚91.97678565979004( proc.memory.availableMB‚ 1314.5234375 disk.nvme1n1.out‚ 35137.03125 disk.md0.out‚ 1423.94140625* proc.memory.percent‚18.171024322509766 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 network.sent‚ 1.9789738e+07" disk.sdb2.out‚1460.09521484375#”_Æ’ù!‚(:‡ Ê·ëÔ½ÄÛÞ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ; H}ú!‚(:ò Ê·ëÔ§ ìß$ disk./.usageGB‚2714.666732788086! disk.sda.out‚1461.89599609375 cpu‚25.226686929355235 disk.rbd2.out‚ 2.74609375 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1461.89599609375 disk.md0.out‚ 1425.6796875 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.97709560394287 disk.rbd2.in‚0! disk.sdb.out‚1461.89599609375" disk.sdb2.out‚1461.89599609375) proc.memory.availableMB‚ 1314.47265625 disk.dm-0.out‚150774.8515625 network.recv‚ 3.023725e+06! disk.nvme0n1.in‚ 6825.97265625 disk.dm-0.in‚ 6924.16796875 disk.nvme1n1.in‚ 98.1953125 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766$ disk.nvme0n1.out‚113199.05859375 disk.md0p1.out‚ 1425.6796875 disk.sda1.out‚0 network.sent‚ 1.9896529e+07( disk./.usagePercent‚75.9032180250497 disk.md0.in‚ 199.59375 disk.rbd0.out‚ 2.74609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚37575.79296875qè´³{û!‚(:ð Ù·ëÔýíºà network.recv‚ 3.028595e+06 disk.sdb1.out‚0 disk.rbd0.out‚ 2.74609375 disk.sda1.out‚0! disk.nvme0n1.in‚ 6825.97265625 disk.md0.in‚ 199.59375 proc.cpu.threads‚17! disk.sdb.out‚1463.28271484375 cpu‚25.22633830038183 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sda2.out‚1463.28271484375 disk.md0.out‚ 1427.01171875! disk.dm-0.out‚151141.11328125 disk.sdb.in‚ 70.9609375" disk.sdb2.out‚1463.28271484375 disk.nvme1n1.in‚ 98.1953125 disk.rbd0.in‚0! disk.sda.out‚1463.28271484375! disk.nvme1n1.out‚ 37807.203125$ memory_percent‚91.97731018066406% proc.memory.availableMB‚ 1314.4375* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚75.93120105561502% disk./.usageGB‚2715.6675415039062 disk.md0p1.in‚ 199.59375 network.sent‚ 1.9902925e+07 disk.rbd2.in‚0$ disk.nvme0n1.out‚113335.16015625 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 disk.rbd2.out‚ 2.74609375 disk.dm-0.in‚ 6924.16796875 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1427.01171875^Ï^¯’ü!‚(:‡ Ù·ëÔ´Ÿ¯à gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.036' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.509 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Þ”ý!‚(:‰ è·ëÔ“ÑÜÞ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t;(1Pþ!‚(:Å è·ëÔ’Ó×à disk.sdb.out‚1464.875 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1464.875 disk.sda.in‚ 128.6328125 proc.cpu.threads‚17$ disk.nvme0n1.out‚114456.29296875 disk.md0.in‚ 199.59375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 2.74609375 network.sent‚ 1.9911321e+07 disk.nvme0n1.in‚ 6828.6171875 disk.sda.out‚1464.875 disk.nvme1n1.in‚ 98.1953125 network.recv‚ 3.032847e+06) disk./.usagePercent‚75.93421506893172 disk.rbd2.out‚ 2.74609375 disk.md0.out‚ 1428.53125 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 disk.rbd0.in‚0 disk.sdb1.in‚0$ memory_percent‚91.97723865509033) proc.memory.availableMB‚ 1314.44921875* proc.memory.percent‚18.171024322509766 cpu‚25.226020378830082% disk./.usageGB‚2715.7753372192383 disk.rbd2.in‚0 disk.dm-0.in‚ 6926.8125! disk.dm-0.out‚152552.79296875 disk.nvme1n1.out‚38096.5 disk.md0p1.out‚ 1428.53125 disk.sda2.out‚1464.875§Š‚Ð2ÿ!- _wandbruntime‚9131  _runtime‚9131}€WzÉ€"à ' lr-AdamW/pg1‚0.00026684286564292636 ' lr-AdamW/pg2‚0.00026684286564292636 trainer/global_step‚7599 % _timestamp‚1.7885337480422432e+09 _runtime‚9131.577748439 _step‚622ƒl^Î"È ' lr-AdamW/pg1‚0.00026684286564292636 ' lr-AdamW/pg2‚0.00026684286564292636 trainer/global_step‚7599 % _timestamp‚1.7885337480422432e+09 _runtime‚9131.577748439 _step‚622î°RÁ•2‚"- _wandbruntime‚9131  _runtime‚9131UÇ“ü'ƒ"öO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚707005503516160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716103745311232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159076722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649550629166080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8677622023630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640839513910784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649549701191680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚623 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8710187280896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159076720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192584251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95315484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192557907 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649550629166080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351634627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141485669 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9098494517248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640839513910784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46170183 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640839531913728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95315486 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚707005479576064 _runtime‚9131.578840284 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711097252352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192557907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9098265735168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711097252352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351660973 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159076722 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141485041 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46169557 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716103998033408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8710187280896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649549701191680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159076720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628295167947770 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619616620909776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚628294242933406 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8678529998634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649549701191680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159076720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pŠÝ%$ool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628295167947770 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619616620909776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚628294242933406 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8678529998634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159076722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649550629166080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8677622023630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640839513910784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649549701191680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚623 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8710187280896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159076720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192584251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95315484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192557907 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649550629166080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351634627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141485669 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9098494517248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640839513910784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46170183 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640839531913728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95315486 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚707005479576064 _runtime‚9131.578840284 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711097252352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192557907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9098265735168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711097252352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351660973 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159076722 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141485041 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46169557 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716103998033408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8710187280896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ï; ¹2…"- _wandbruntime‚9132  _runtime‚9132Yx'†"ÿM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159096880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640921150999552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640921150999552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8678739296908 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8711308741120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚707095705280512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192609108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716195390041088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351705990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351679618 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141502935 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 $ _timestamp‚1.788533749000867e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192582738 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649633369725440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8679647272120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46176006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46175386 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192609108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚9132.536931941 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159096882 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159096880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141503557 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351679618 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619695638504320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚707095729220608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351705990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351679618 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141502935 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 $ _timestamp‚1.788533749000867e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192582738 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649633369725440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8679647272120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46176006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46175386 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192609108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚9132.536931941 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159096882 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159096880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141503557 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351679618 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619695638504320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚707095729220608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640921150999552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8678739296908 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8711308741120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚707095705280512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192609108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716195390041088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 Pðx’"‚(:í ¤¸ëÔà†ôß cpu‚25.224741597701865 disk.sda1.in‚0 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 98.1953125 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 6927.15625" disk.sdb2.out‚1471.30712890625$ disk./.usageGB‚2721.944263458252! disk.nvme0n1.out‚ 116164.21875! disk.dm-0.out‚160456.08984375$ memory_percent‚91.97924137115479 network.sent‚ 2.0034618e+07 network.recv‚ 3.051188e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1434.6796875! disk.sdb.out‚1471.30712890625 disk.sdb2.in‚ 70.9609375) disk./.usagePercent‚76.10670082847075! disk.sda.out‚1471.30712890625 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1314.12109375! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766# disk.nvme1n1.out‚44291.87109375 disk.md0p1.out‚ 1434.6796875 disk.nvme0n1.in‚ 6828.9609375 disk.sdb1.in‚0 disk.rbd2.out‚ 2.74609375" disk.sda2.out‚1471.30712890625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.59375 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375×Þd2“"- _wandbruntime‚9184  _runtime‚9184£«âÈ”" ' lr-AdamW/pg1‚0.00026176843275539874 ' lr-AdamW/pg2‚0.00026176843275539874 trainer/global_step‚7649 $ _timestamp‚1.788533801283439e+09 _runtime‚9184.819116261 _step‚626üŸú Í•"Ç ' lr-AdamW/pg1‚0.00026176843275539874 ' lr-AdamW/pg2‚0.00026176843275539874 trainer/global_step‚7649 $ _timestamp‚1.788533801283439e+09 _runtime‚9184.819116261 _step‚626òŒNu»2–"- _wandbruntime‚9184  _runtime‚9184Å>© ü'—"öO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8766274705920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160084720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644920486204928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644920468201984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653686742907904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8767184677376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142377589 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353887529 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193829153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚7649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95918736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46459481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711516788738560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160084722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8767184677376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193829153 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚720673783586816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8733499278738 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653686742907904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8767184677376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142377589 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353887529 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193829153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚7649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95918736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46459481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711516788738560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160084722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8767184677376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193829153 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚720673783586816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8733499278738 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353958892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142395460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95930801 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632382059798108 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353932520 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193827640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚711607014ŠL443008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8734616552016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711606990502912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9158166150656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193827640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46465305 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353958892 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚653770411441664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚653769483453952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8735524527228 _runtime‚9185.796113917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193854010 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353932520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95930799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885338022600682e+09 '›"…N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚720765180593664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193827640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚711607014443008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8734616552016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711606990502912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9158166150656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193827640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46465305 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353958892 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚653770411441664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚653769483453952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8735524527228 _runtime‚9185.796113917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193854010 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353932520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95930799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885338022600682e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8767396166144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚720764929981952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9157939479040 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160104880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353958892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142395460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95930801 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632382059798108 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353932520 Eœ”¡"‚(:‰ ¸ëÔáŹà gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{‹¦_¢"‚(:Ô ¸ëÔšÿÇá) disk./.usagePercent‚76.12279259629142 disk.md0.in‚ 199.59375 disk.sdb2.out‚ 1474.921875 disk.sda1.out‚0' proc.memory.availableMB‚ 1314.578125 network.recv‚ 3.060459e+06% disk./.usageGB‚2722.5197830200195 disk.sda.in‚ 128.6328125 disk.md0.out‚ 1438.171875 disk.dm-0.in‚ 6971.328125 disk.sda.out‚ 1474.921875 disk.sdb1.out‚0# memory_percent‚91.9764518737793* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 98.1953125 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 6873.1328125 disk.sdb.out‚ 1474.921875 network.sent‚ 2.0149829e+07 disk.sda2.in‚ 128.6328125! disk.dm-0.out‚163291.57421875 disk.md0p1.in‚ 199.59375 cpu‚25.22405609380733 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.out‚ 117857.8125 disk.rbd0.in‚0! proc.memory.rssMB‚ 2977.140625 disk.rbd2.in‚0 disk.sda2.out‚ 1474.921875# disk.nvme1n1.out‚45433.76171875 disk.md0p1.out‚ 1438.171875 proc.cpu.threads‚17’61Д£"‚(:‰ ѸëÔÏÞüÞ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ð(c¤"‚(:Ø ѸëÔºœöß disk.sdb1.in‚0 disk.md0.in‚ 199.59375 disk.sdb.out‚ 1476.59375 disk.sda.in‚ 128.6328125 proc.cpu.threads‚17 disk.md0p1.in‚ 199.59375 disk.sda2.out‚ 1476.59375$ memory_percent‚91.97800159454346 disk.sdb2.in‚ 70.9609375 network.recv‚ 3.06527e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 cpu‚25.223756128593376) disk./.usagePercent‚76.11471487421225 disk.md0p1.out‚ 1439.78515625 disk.rbd0.out‚ 2.74609375 disk.rbd2.in‚0! disk.dm-0.out‚163845.15234375 disk.sda.out‚ 1476.59375* proc.memory.percent‚18.171024322509766$ disk./.usageGB‚2722.230884552002# disk.nvme1n1.out‚45866.41796875 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1476.59375 network.sent‚ 2.0153875e+07 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 6873.1328125" disk.nvme0n1.out‚ 117978.734375 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 disk.nvme1n1.in‚ 98.1953125 disk.sda1.in‚0 disk.md0.out‚ 1439.78515625 disk.dm-0.in‚ 6971.328125) proc.memory.availableMB‚ 1314.32421875i`›Í2¥"- _wandbruntime‚9239  _runtime‚9239܂ߩɦ"à _step‚630 ' lr-AdamW/pg1‚0.00025671570067569723 ' lr-AdamW/pg2‚0.00025671570067569723 trainer/global_step‚7699 % _timestamp‚1.7885338555593567e+09 _runtime‚9239.094935819­{_Χ"È trainer/global_step‚7699 % _timestamp‚1.7885338555593567e+09 _runtime‚9239.094935819 _step‚630 ' lr-AdamW/pg1‚0.00025671570067569723 ' lr-AdamW/pg2‚0.00025671570067569723öë”׳2¨"- _wandbruntime‚9239  _runtime‚9239È% Fü'©"öO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143271386 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143270771 trainer/global_step‚7699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649001422493184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8823272102400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚716028050020864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649001422493184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161092720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚716028073960960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161092720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 356140431 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8822362130944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195074055 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46749400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657823784624128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657823784624128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9215322717184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649001440496128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46748787 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195047711 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8790284508850 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195047711 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161092722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96521986 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161092722 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8823272102400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96521984 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161092720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚716028073960960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161092720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 356140431 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8822362130944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195074055 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46749400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657823784624128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657823784624128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9215322717184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649001440496128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46748787 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195047711 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8790284508850 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195047711 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161092722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96521986 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161092722 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8823272102400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96521984 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649001422493184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8823272102400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚716028050020864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649001422493184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249÷pÓ­”«"‚(:‰ à¸ëÔ»è¡à gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]rö/|¬"‚(:ñ à¸ëÔçùÏà" disk.sda2.out‚1478.03173828125) disk./.usagePercent‚76.05720564850286! disk.sdb.out‚1478.03173828125 disk.sda.in‚ 128.6328125! disk.sda.out‚1478.03173828125 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0 disk.rbd2.in‚0" disk.sdb2.out‚1478.03173828125 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0$ memory_percent‚91.97797775268555* proc.memory.percent‚18.171024322509766# disk.nvme1n1.out‚46411.01953125 network.sent‚ 2.0160018e+07% disk./.usageGB‚2720.1740760803223 disk.sdb.in‚ 70.9609375 disk.sda1.in‚0 disk.dm-0.in‚ 6987.0546875 disk.dm-0.out‚165014.8515625 network.recv‚ 3.069645e+06 disk.nvme1n1.in‚ 98.1953125 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375 disk.nvme0n1.in‚ 6888.859375 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1441.16796875 cpu‚25.223443848344907 proc.cpu.threads‚17 disk.md0p1.out‚ 1441.16796875 disk.rbd0.in‚0 disk.rbd2.out‚ 2.74609375$ disk.nvme0n1.out‚118606.83203125 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2977.140625' proc.memory.availableMB‚ 1314.328125¸HºË2­"- _wandbruntime‚9240  _runtime‚92406Ž—E'®"€N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9216296605696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195098912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649083059581952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627597397958720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161112880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649083059581952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657907453157888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 trainer/global_step‚7699 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649083041579008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143289181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657907453157888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8823483591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716118299665408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627597414998080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8791401782336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356211794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46754518 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195072542 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143288567 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96534051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725334808262656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649083041579008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161112882 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8824393575936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649083059581952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657907453157888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 trainer/global_step‚7699 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649083041579008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143289181 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657907453157888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8823483591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716118299665408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627597414998080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8791401782336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356211794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46754518 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195072542 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143288567 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96534051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725334808262656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649083041579008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161112882 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8824393575936 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640315438894694 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162100720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47039020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661961754314752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653082394787328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8879359527424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8878449555968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9273912813568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358419679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653082394787328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661960826340352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196318957 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661960826340352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661961754314752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720539335243264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9273681934336 _step‚635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720539359183360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631469277130736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360s¸(¾"úO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8879359527424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8878449555968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9273912813568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358419679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653082394787328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661960826340352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196318957 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661960826340352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661961754314752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720539335243264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9273681934336 _step‚635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720539359183360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631469277130736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358393333 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196292613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162100720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144164873 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640315438894694 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162100720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47039020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661961754314752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653082394787328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0û•~ç›r¿"‚(:ç œ¹ëÔù¨à network.sent‚ 2.0365738e+07 disk.md0.in‚ 199.59375 proc.cpu.threads‚17 disk.sda2.out‚1485.056640625 network.recv‚ 3.088901e+06 disk.nvme1n1.in‚ 98.1953125 disk.md0p1.out‚ 1447.9453125 disk.sdb1.in‚0) disk./.usagePercent‚76.04494491367494 disk.sda.out‚1485.056640625# disk.nvme1n1.out‚47513.62890625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.dm-0.out‚ 168666.4375 disk.rbd0.out‚ 2.74609375 disk.md0.out‚ 1447.9453125! proc.memory.rssMB‚ 2977.140625 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6954.98046875 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚1485.056640625$ memory_percent‚91.97835922241211 disk.sda2.in‚ 128.6328125 disk.sdb.out‚1485.056640625% disk./.usageGB‚2719.7355728149414 disk.sda.in‚ 128.6328125 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375' proc.memory.availableMB‚ 1314.265625 cpu‚25.222088199307354 disk.rbd2.in‚0 disk.dm-0.in‚ 7053.17578125* proc.memory.percent‚18.171024322509766$ disk.nvme0n1.out‚121152.80859375 disk.sdb2.in‚ 70.9609375tì‰Æ“À"‚(:ˆ œ¹ëÔÒ¤›á gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0홳¿2Á"- _wandbruntime‚9300  _runtime‚9300Á0©‘-Â"€N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97137299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9274873392128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47044792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144182696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97137301 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640394648748152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196343814 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653164013873152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196343814 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358438324 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144182091 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162120880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196317444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846371062232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162120882 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358464696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚729904672368640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720629584887808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720629560947712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653163995870208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8880481000960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162120882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8847279037444 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚7749 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640395573762724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358464696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8879571016192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚9300.294812053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358438324 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653164013873152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196343814 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358438324 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144182091 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162120880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196317444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846371062232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162120882 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358464696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚729904672368640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720629584887808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720629560947712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653163995870208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8880481000960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162120882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8847279037444 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚7749 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640395573762724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358464696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8879571016192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚9300.294812053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358438324 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FÎÑ"È ' lr-AdamW/pg1‚0.00024667954612508496 ' lr-AdamW/pg2‚0.00024667954612508496 trainer/global_step‚7799 % _timestamp‚1.7885339785342805e+09 _runtime‚9362.070073893 _step‚638þ‡2é2Ò"- _wandbruntime‚9362  _runtime‚9362EAž“Ó"õO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9332251250688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666098796030976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885339785356941e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666098796030976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47329720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360672581 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47330326 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚7799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145058812 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163108722 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97728484 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644321270862638 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734382871716352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657163331075584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666097868056576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197537515 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97728486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734383111855616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8935446952448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197563859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725050620465664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902039019066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8901131044062 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666097868056576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163108722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163108720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _step‚639­y„~(Ô"úO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360672581 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47330326 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚7799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145058812 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163108722 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97728484 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644321270862638 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734382871716352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657163331075584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666097868056576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197537515 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97728486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734383111855616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8935446952448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197563859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725050620465664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902039019066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8901131044062 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666097868056576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163108722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163108720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _step‚639 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9332251250688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666098796030976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885339785356941e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666098796030976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47329720ÿ/M8á2Õ"- _wandbruntime‚9363  _runtime‚9363-q :'Ö"ÿM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666180608602624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97740551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644401405730460 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657244968164352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725140870110208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666181536590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9333647818752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635499174452480 _step‚640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657244950161408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197562346 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145076013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8902248317340 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47335464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360691226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734474263122944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360717598 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163128880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8936568425984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360691226 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163128880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370æ`GÄ>%×"„N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163128880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8936568425984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360691226 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163128880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666180608602624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97740551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644401405730460 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657244968164352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725140870110208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666181536590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9333647818752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635499174452480 _step‚640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657244950161408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197562346 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145076013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8902248317340 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47335464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360691226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734474263122944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360717598 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725€­¹tŸ2Ø"- _wandbruntime‚9363  _runtime‚9363hÛ†Ú`Ù"Ú 1loss/FBt4/assignment_loss‚0.29064474941945945 0loss/FRt3/assignment_loss‚0.8556018934438103 2loss/SRqqt4/assignment_loss‚1.4081747456852878 0loss/FBt2/assignment_loss‚0.2067640369248657 /loss/FRt1/detection_loss‚3.3832542300224304 _runtime‚9363.170928697 trainer/global_step‚7799 /loss/FRt2/assignment_loss‚2.910468153655529 2loss/SRqqt1/assignment_loss‚1.5155234021299027 .loss/FRt4/detection_loss‚2.100900265574455 1loss/SRqqt3/detection_loss‚0.7229348327964544 _step‚641 /loss/FRt2/detection_loss‚2.5271567869970095 1loss/SRqqt1/detection_loss‚2.8329461304580463 % _timestamp‚1.7885339796356487e+09 0loss/FRt1/assignment_loss‚3.2194674809773765 1loss/FBt3/assignment_loss‚0.07144422282354791 /loss/FBt1/detection_loss‚2.1519538050606135 /loss/FRt3/detection_loss‚1.2082635257393122 epoch‚6 0loss/FRt4/assignment_loss‚3.2921274590492247 2loss/SRqqt2/assignment_loss‚1.1235733343095615 1loss/SRqqt2/detection_loss‚2.2608379485278296 /loss/FBt3/detection_loss‚0.4854136405873202 1loss/FBt1/assignment_loss‚0.22333720763474826 1loss/SRqqt4/detection_loss‚2.1880092058862957 /loss/FBt2/detection_loss‚1.9471041826825393 /loss/FBt4/detection_loss‚2.3128795653581617 %loss/total_loss‚40.38053523612312 2loss/SRqqt3/assignment_loss‚1.1417544303793397ï ›‰eÚ"ß /loss/FRt3/detection_loss‚1.2082635257393122 epoch‚6 0loss/FRt4/assignment_loss‚3.2921274590492247 2loss/SRqqt2/assignment_loss‚1.1235733343095615 1loss/SRqqt2/detection_loss‚2.2608379485278296 /loss/FBt3/detection_loss‚0.4854136405873202 1loss/FBt1/assignment_loss‚0.22333720763474826 1loss/SRqqt4/detection_loss‚2.1880092058862957 /loss/FBt2/detection_loss‚1.9471041826825393 /loss/FBt4/detection_loss‚2.3128795653581617 %loss/total_loss‚40.38053523612312 2loss/SRqqt3/assignment_loss‚1.1417544303793397 1loss/FBt4/assignment_loss‚0.29064474941945945 0loss/FRt3/assignment_loss‚0.8556018934438103 2loss/SRqqt4/assignment_loss‚1.4081747456852878 0loss/FBt2/assignment_loss‚0.2067640369248657 /loss/FRt1/detection_loss‚3.3832542300224304 _runtime‚9363.170928697 trainer/global_step‚7799 /loss/FRt2/assignment_loss‚2.910468153655529 2loss/SRqqt1/assignment_loss‚1.5155234021299027 .loss/FRt4/detection_loss‚2.100900265574455 1loss/SRqqt3/detection_loss‚0.7229348327964544 _step‚641 /loss/FRt2/detection_loss‚2.5271567869970095 1loss/SRqqt1/detection_loss‚2.8329461304580463 % _timestamp‚1.7885339796356487e+09 0loss/FRt1/assignment_loss‚3.2194674809773765 1loss/FBt3/assignment_loss‚0.07144422282354791 /loss/FBt1/detection_loss‚2.1519538050606135ÏŒ–”Û"‚(:‰ ç¹ëÔãêµá gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Q\€Ü"‚(:õ ç¹ëÔ£¿•â network.sent‚ 2.0575991e+07 disk.nvme1n1.in‚ 98.1953125 disk.rbd0.in‚0" disk.sda2.out‚1493.52197265625 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1493.52197265625 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375! disk.nvme0n1.in‚ 6954.98046875# disk.nvme0n1.out‚121728.4921875) disk./.usagePercent‚76.06769848501087" disk.sdb2.out‚1493.52197265625 disk.sda.in‚ 128.6328125# disk.nvme1n1.out‚47867.26953125 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 network.recv‚ 3.110925e+06% disk./.usageGB‚2720.5493507385254 disk.sdb1.in‚0 disk.md0.in‚ 199.59375 disk.rbd2.in‚0 disk.md0p1.out‚ 1456.12890625 disk.sdb1.out‚0 disk.sda1.in‚0* proc.memory.percent‚18.171024322509766 cpu‚25.219959780734705! disk.sdb.out‚1493.52197265625 disk.md0.out‚ 1456.12890625 disk.dm-0.in‚ 7053.17578125! disk.dm-0.out‚169595.76171875$ memory_percent‚91.98698997497559( proc.memory.availableMB‚ 1312.8515625 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 2.74609375W&å^”Ý"‚(:‰ ö¹ëÔàÀÀà gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C2h…|Þ"‚(:ñ ö¹ëÔÕ–¥á network.sent‚ 2.0685821e+07 disk.sda1.in‚0 disk.md0.out‚ 1457.515625 disk.dm-0.in‚ 7053.17578125) proc.memory.availableMB‚ 1312.80859375 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚121832.1484375 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1494.96826171875! disk.nvme0n1.in‚ 6954.98046875 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 98.1953125# disk.nvme1n1.out‚47988.23046875# memory_percent‚91.9872522354126 network.recv‚ 3.115938e+06% disk./.usageGB‚2720.5494079589844 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1494.96826171875 disk.md0p1.out‚ 1457.515625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06770008491885 disk.rbd0.in‚0 disk.rbd2.out‚ 2.74609375! disk.dm-0.out‚169820.37890625 disk.sdb.in‚ 70.9609375 cpu‚25.219561102019842 disk.rbd0.out‚ 2.74609375 disk.md0.in‚ 199.59375! disk.sdb.out‚1494.96826171875 disk.md0p1.in‚ 199.59375 proc.cpu.threads‚17" disk.sdb2.out‚1494.96826171875넼֓ß"‚(:ˆ …ºëÔžö˜à gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0llÖmà"‚(:â …ºëÔøÈóà disk.dm-0.in‚ 7053.17578125$ disk./.usageGB‚2720.274574279785 disk.md0p1.in‚ 199.59375 disk.nvme1n1.out‚ 48052.65625 network.recv‚ 3.12073e+06 disk.rbd0.in‚0 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2977.140625 disk.sdb1.in‚0 disk.rbd2.out‚ 2.74609375 disk.sda.out‚ 1496.78515625 disk.nvme1n1.in‚ 98.1953125( disk./.usagePercent‚76.0600156202216 disk.rbd2.in‚0! disk.nvme0n1.in‚ 6954.98046875$ disk.nvme0n1.out‚121858.94140625$ memory_percent‚91.98727607727051 disk.md0p1.out‚ 1459.2734375! disk.dm-0.out‚169900.59765625 disk.sdb.out‚ 1496.78515625 disk.sdb2.in‚ 70.9609375( proc.memory.availableMB‚ 1312.8046875* proc.memory.percent‚18.171024322509766 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.219160960721073 disk.rbd0.out‚ 2.74609375 disk.md0.out‚ 1459.2734375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 proc.cpu.threads‚17 disk.sdb2.out‚ 1496.78515625 network.sent‚ 2.0689801e+07 disk.sda2.in‚ 128.6328125 disk.sda2.out‚ 1496.78515625%–yÊ’á"‚(:‡ ”ºëÔùâ·à gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À(‡vâ"‚(:ë ”ºëÔÿ›òà! disk.nvme0n1.in‚ 6954.98046875 disk.dm-0.in‚ 7053.17578125 disk.md0p1.in‚ 199.59375 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.sdb.out‚1498.2861328125! disk.sdb2.out‚1498.2861328125 disk.md0p1.out‚ 1460.71875 disk.sdb1.out‚0 network.sent‚ 2.0698165e+07% disk./.usageGB‚2720.2355003356934 disk.sda.out‚1498.2861328125 disk.nvme1n1.in‚ 98.1953125# disk.nvme1n1.out‚48052.81640625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 1460.71875 network.recv‚ 3.125135e+06 disk.rbd2.out‚ 2.74609375$ memory_percent‚91.99025630950928! proc.memory.rssMB‚ 2977.140625 disk.dm-0.out‚170092.8203125 disk.sdb2.in‚ 70.9609375! disk.sda2.out‚1498.2861328125) disk./.usagePercent‚76.05892309639108 disk.sdb.in‚ 70.9609375 disk.rbd0.out‚ 2.74609375) proc.memory.availableMB‚ 1312.31640625* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17$ disk.nvme0n1.out‚122040.00390625 disk.sda1.in‚0 cpu‚25.21875995842008 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sda1.out‚0K˜o2ã"- _wandbruntime‚9424  _runtime‚9424~è­‰Éä"à ' lr-AdamW/pg1‚0.00024169821517374012 ' lr-AdamW/pg2‚0.00024169821517374012 trainer/global_step‚7849 % _timestamp‚1.7885340413619606e+09 _runtime‚9424.897544424 _step‚642HK»Îå"È trainer/global_step‚7849 % _timestamp‚1.7885340413619606e+09 _runtime‚9424.897544424 _step‚642 ' lr-AdamW/pg1‚0.00024169821517374012 ' lr-AdamW/pg2‚0.00024169821517374012‚ׂ3°2æ"- _wandbruntime‚9424  _runtime‚9424%ëDü'ç"öO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639371019545776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8957008299170 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8991534377472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145950770 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚729561929628160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670235837747200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670235837747200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198782417 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145951399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚648328027844946 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚661244285366784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639371036585136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164116720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661244303369728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8990624406016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198808761 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885340413630428e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198808761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚9424.898619649 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670234909772800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8990624406016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚738952869760000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚738952625426432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729561905688064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8991534377472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98331734 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚643 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164116722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9390940131840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚648328952859310 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164116720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362925483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164116722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47619663 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198782417 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661244303369728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670234909772800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚661244285366784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98331736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9390719738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8957916274174 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362899137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615>î(è"ûO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670234909772800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8990624406016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚738952869760000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚738952625426432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729561905688064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8991534377472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98331734 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚643 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164116722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9390940131840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚648328952859310 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164116720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362925483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164116722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47619663 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198782417 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661244303369728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670234909772800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚661244285366784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98331736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9390719738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8957916274174 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362899137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639371019545776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8957008299170 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8991534377472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145950770 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚729561929628160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670235837747200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670235837747200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198782417 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145951399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚648328027844946 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚661244285366784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639371036585136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164116720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661244303369728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8990624406016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198808761 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885340413630428e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198808761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚9424.898619649ƒ’9~/2é"- _wandbruntime‚9426  _runtime‚9426þ<ÕÏ[ê"ÿM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164136880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚661325922455552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145968619 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 9426.07196646 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362944128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661325904452608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198833618 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚648409087727340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661325904452608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9391890800128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164136882 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198833618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362944128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362970500 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639450037140320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚648408162712768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639450054179680 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98343799 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚7849 % _timestamp‚1.7885340425352402e+09 _step‚644 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47625440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670317650318848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164136882 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47624820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚739044272804352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8958125572448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164136880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚729652155332608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145968619 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 9426.07196646 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362944128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661325904452608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198833618 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚648409087727340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661325904452608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9391890800128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164136882 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198833618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362944128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362970500 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639450037140320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚648408162712768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639450054179680 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98343799 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚7849 % _timestamp‚1.7885340425352402e+09 _step‚644„ Õð2ì"- _wandbruntime‚9426  _runtime‚9426:1Ü“_í"Ù _runtime‚9426.072939275 0loss/FRt4/assignment_loss‚2.9863910724719367 1loss/SRqqt1/detection_loss‚2.9783412762310193 1loss/SRqqt4/detection_loss‚2.2360701067373157 -loss/FBt4/detection_loss‚2.47457624398745 % _timestamp‚1.7885340425376022e+09 /loss/FRt1/detection_loss‚2.8275675036405263 1loss/SRqqt3/detection_loss‚0.9297787419706584 /loss/FBt2/detection_loss‚1.9468031972646713 2loss/SRqqt4/assignment_loss‚1.4524614377878606 /loss/FRt2/detection_loss‚2.3402426440100337 /loss/FRt4/detection_loss‚2.0393342388172946 &loss/total_loss‚40.180331141719805 epoch‚6 2loss/SRqqt2/assignment_loss‚1.5198886493413613 2loss/SRqqt3/assignment_loss‚0.9109261792086181 0loss/FBt4/assignment_loss‚0.1381869385025381 trainer/global_step‚7849 2loss/SRqqt1/assignment_loss‚1.2984513050836066 1loss/SRqqt2/detection_loss‚2.0111563380422264 /loss/FBt3/detection_loss‚0.7085235279558574 2loss/FBt1/assignment_loss‚0.056769585252709476 /loss/FRt3/detection_loss‚1.4027798975714378 .loss/FBt1/detection_loss‚2.551081556540269 _step‚645 /loss/FRt3/assignment_loss‚1.253846301800675 1loss/FBt3/assignment_loss‚0.11266600321054404 /loss/FRt1/assignment_loss‚3.290472957648729 0loss/FRt2/assignment_loss‚2.6710524467321544 1loss/FBt2/assignment_loss‚0.04296299191030641»Xƒndî"Þ 2loss/SRqqt1/assignment_loss‚1.2984513050836066 1loss/SRqqt2/detection_loss‚2.0111563380422264 /loss/FBt3/detection_loss‚0.7085235279558574 2loss/FBt1/assignment_loss‚0.056769585252709476 /loss/FRt3/detection_loss‚1.4027798975714378 .loss/FBt1/detection_loss‚2.551081556540269 _step‚645 /loss/FRt3/assignment_loss‚1.253846301800675 1loss/FBt3/assignment_loss‚0.11266600321054404 /loss/FRt1/assignment_loss‚3.290472957648729 0loss/FRt2/assignment_loss‚2.6710524467321544 1loss/FBt2/assignment_loss‚0.04296299191030641 _runtime‚9426.072939275 0loss/FRt4/assignment_loss‚2.9863910724719367 1loss/SRqqt1/detection_loss‚2.9783412762310193 1loss/SRqqt4/detection_loss‚2.2360701067373157 -loss/FBt4/detection_loss‚2.47457624398745 % _timestamp‚1.7885340425376022e+09 /loss/FRt1/detection_loss‚2.8275675036405263 1loss/SRqqt3/detection_loss‚0.9297787419706584 /loss/FBt2/detection_loss‚1.9468031972646713 2loss/SRqqt4/assignment_loss‚1.4524614377878606 /loss/FRt2/detection_loss‚2.3402426440100337 /loss/FRt4/detection_loss‚2.0393342388172946 &loss/total_loss‚40.180331141719805 epoch‚6 2loss/SRqqt2/assignment_loss‚1.5198886493413613 2loss/SRqqt3/assignment_loss‚0.9109261792086181 0loss/FBt4/assignment_loss‚0.1381869385025381 trainer/global_step‚7849…Óg“ï"‚(:ˆ £ºëÔ²þ¨à gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôLÞøzð"‚(:ï £ºëÔÀ«Ðá disk.sdb2.in‚ 70.9609375 disk.md0.out‚ 1462.10546875 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125 network.sent‚ 2.0714237e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 2.74609375 disk.rbd0.in‚0$ memory_percent‚91.97807312011719 network.recv‚ 3.129182e+06" disk.sdb2.out‚1499.72900390625! proc.memory.rssMB‚ 2977.140625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 6954.98046875! disk.sdb.out‚1499.72900390625 disk.sda1.in‚0$ disk.nvme0n1.out‚122259.17578125 disk.dm-0.in‚ 7053.17578125 disk.nvme1n1.in‚ 98.1953125# disk.nvme1n1.out‚48053.10546875 disk.md0p1.in‚ 199.59375 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1462.10546875" disk.sda2.out‚1499.72900390625) disk./.usagePercent‚76.06066123642255 disk.rbd0.out‚ 2.74609375 cpu‚25.218356083837573$ disk./.usageGB‚2720.297664642334 disk.dm-0.out‚ 170312.28125! disk.sda.out‚1499.72900390625% proc.memory.availableMB‚ 1314.3125* proc.memory.percent‚18.171024322509766oö“ñ"‚(:ˆ ²ºëÔçðêà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ_“jò"‚(:ß ²ºëÔ¹¦ƒá disk.rbd2.out‚ 2.74609375! proc.memory.rssMB‚ 2977.140625 disk.sda1.out‚0! disk.nvme0n1.in‚ 6954.98046875 disk.nvme0n1.out‚ 122370.875$ memory_percent‚91.97824001312256) disk./.usagePercent‚76.03309247536397 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.sdb.out‚ 1501.6953125 disk.rbd0.out‚ 2.74609375 disk.sda2.out‚ 1501.6953125 disk.dm-0.in‚ 7053.17578125 disk.sdb2.out‚ 1501.6953125 network.sent‚ 2.0811232e+07 network.recv‚ 3.133944e+06 disk.md0p1.in‚ 199.59375! disk.dm-0.out‚170424.13671875 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1464.0078125 disk.md0.out‚ 1464.0078125 disk.sda.in‚ 128.6328125) proc.memory.availableMB‚ 1314.28515625% disk./.usageGB‚2719.3116722106934 disk.sda.out‚ 1501.6953125* proc.memory.percent‚18.171024322509766 cpu‚25.21798732354154 disk.nvme1n1.in‚ 98.1953125# disk.nvme1n1.out‚48053.26171875 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375 disk.sdb1.in‚0 disk.rbd2.in‚0TZñw”ó"‚(:‰ ÁºëÔÖªµà gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i^yÂvô"‚(:ë ÁºëÔ¯Ïá network.recv‚ 3.138675e+06) disk./.usagePercent‚76.03834497326831 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 network.sent‚ 2.081737e+07 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1503.0751953125! disk.sdb2.out‚1503.0751953125 disk.sda.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.md0.out‚ 1465.3203125 disk.md0p1.out‚ 1465.3203125 disk.rbd2.out‚ 2.74609375 disk.md0p1.in‚ 199.59375 disk.sda2.in‚ 128.6328125! disk.sda2.out‚1503.0751953125$ memory_percent‚91.97824001312256 disk.dm-0.in‚ 7053.17578125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1314.28515625 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6954.98046875$ disk.nvme0n1.out‚122438.76953125 disk.dm-0.out‚ 170492.34375 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚48053.57421875 disk.rbd0.out‚ 2.74609375* proc.memory.percent‚18.171024322509766$ disk./.usageGB‚2719.499526977539 disk.sda.out‚1503.0751953125 disk.nvme1n1.in‚ 98.1953125! proc.memory.rssMB‚ 2977.140625 cpu‚25.21753683395514® ¡Z2õ"- _wandbruntime‚9479  _runtime‚9479]¢³Çö"Á _step‚646 & lr-AdamW/pg1‚0.0002367427676334833 & lr-AdamW/pg2‚0.0002367427676334833 trainer/global_step‚7899 % _timestamp‚1.7885340964125352e+09 _runtime‚9479.9481603389+»Ì÷"Æ trainer/global_step‚7899 % _timestamp‚1.7885340964125352e+09 _runtime‚9479.948160338 _step‚646 & lr-AdamW/pg1‚0.0002367427676334833 & lr-AdamW/pg2‚0.0002367427676334833†,Ôí$2ø"- _wandbruntime‚9479  _runtime‚9479îÜù"öO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146842693 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200053663 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674371951489024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674371951489024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47907709 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665325257660928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDevicÖéL eStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743522528518144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652335709841618 trainer/global_step‚7899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200027319 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚647 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674372879463424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365152039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9047621802496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146843308 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615³Ý\(ú"ûO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674372879463424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365152039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9047621802496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146843308 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146842693 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200053663 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674371951489024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674371951489024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47907709 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665325257660928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743522528518144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652335709841618 trainer/global_step‚7899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200027319 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚647 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0‡šJcžû"‚(:“ кëÔ®˜­à gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H6ÔÛ}ü"‚(:ò кëÔñóá disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.nvme1n1.in‚ 98.1953125 disk.rbd2.in‚0$ disk.nvme0n1.out‚122574.18359375 network.recv‚ 3.142767e+06) disk./.usagePercent‚76.03832918750956 disk.rbd0.in‚0 disk.sda.in‚ 128.6328125$ memory_percent‚91.97962284088135 disk.md0p1.out‚ 1466.859375! proc.memory.rssMB‚ 2977.140625" disk.sdb2.out‚1504.68115234375 network.sent‚ 2.0834198e+07 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0! disk.sdb.out‚1504.68115234375) proc.memory.availableMB‚ 1314.05859375 disk.md0.out‚ 1466.859375 disk.dm-0.in‚ 7053.17578125! disk.dm-0.out‚170627.92578125! disk.sda.out‚1504.68115234375" disk.nvme1n1.out‚ 48053.7421875 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 cpu‚25.217110292861438! disk.nvme0n1.in‚ 6954.98046875% disk./.usageGB‚2719.4989624023438 disk.sdb1.out‚0" disk.sda2.out‚1504.68115234375 disk.sdb.in‚ 70.9609375рˣ2ý"- _wandbruntime‚9481  _runtime‚9481Ë®b«'þ"ÿM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165144880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9048743276032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200078520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200052150 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9014910802768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9014002827556 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743614174370304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365197030 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146860542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98947051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365223402 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9450733815296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 9481.26590746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98947049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9450505046528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652415844709648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165144880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643400933906880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165144882 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665406876746752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9048743276032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665406876746752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637ßy¸j\ÿ"„N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365197030 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146860542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98947051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365223402 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9450733815296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 9481.26590746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98947049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9450505046528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652415844709648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165144880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643400933906880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165144882 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665406876746752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9048743276032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665406876746752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165144880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9048743276032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200078520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200052150 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9014910802768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9014002827556 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743614174370304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ˆ~ë2€#- _wandbruntime‚9481  _runtime‚9481W*NW#Ñ 0loss/SRqqt2/assignment_loss‚1.40814095597182 1loss/FBt4/assignment_loss‚0.22007441355322044 0loss/SRqqt2/detection_loss‚2.162954281909125 .loss/FBt4/detection_loss‚2.975407307346662 0loss/FRt1/assignment_loss‚2.7843138122558595 0loss/FRt3/assignment_loss‚1.0650314793867224 2loss/SRqqt4/assignment_loss‚1.4099176686257124 1loss/FBt2/assignment_loss‚0.19893990835055758 0loss/SRqqt1/detection_loss‚2.527752883732319 2loss/SRqqt3/assignment_loss‚0.9571904534056226 /loss/FRt4/detection_loss‚2.3179103292447203 0loss/SRqqt4/detection_loss‚2.050698620743222 %loss/total_loss‚39.98379332291462 .loss/FRt1/detection_loss‚3.104169635772705 /loss/FRt2/detection_loss‚2.3073910200024588 /loss/FRt4/assignment_loss‚2.938301347196102 1loss/SRqqt1/assignment_loss‚1.392855716869235 1loss/SRqqt3/detection_loss‚0.8388997120695694 .loss/FBt1/detection_loss‚2.332393386147239 /loss/FBt2/detection_loss‚2.3537337596599874 /loss/FBt3/detection_loss‚0.4320072028785944 _runtime‚9481.267078635 .loss/FRt2/assignment_loss‚2.61881822177342 1loss/FBt1/assignment_loss‚0.14710658783389424 /loss/FRt3/detection_loss‚1.3865233521689386 trainer/global_step‚7899 % _timestamp‚1.7885340977316582e+09 _step‚649 1loss/FBt3/assignment_loss‚0.05326126601691943 epoch‚6f#§@\‚#Ö _runtime‚9481.267078635 .loss/FRt2/assignment_loss‚2.61881822177342 1loss/FBt1/assignment_loss‚0.14710658783389424 /loss/FRt3/detection_loss‚1.3865233521689386 trainer/global_step‚7899 % _timestamp‚1.7885340977316582e+09 _step‚649 1loss/FBt3/assignment_loss‚0.05326126601691943 epoch‚6 0loss/SRqqt2/assignment_loss‚1.40814095597182 1loss/FBt4/assignment_loss‚0.22007441355322044 0loss/SRqqt2/detection_loss‚2.162954281909125 .loss/FBt4/detection_loss‚2.975407307346662 0loss/FRt1/assignment_loss‚2.7843138122558595 0loss/FRt3/assignment_loss‚1.0650314793867224 2loss/SRqqt4/assignment_loss‚1.4099176686257124 1loss/FBt2/assignment_loss‚0.19893990835055758 0loss/SRqqt1/detection_loss‚2.527752883732319 2loss/SRqqt3/assignment_loss‚0.9571904534056226 /loss/FRt4/detection_loss‚2.3179103292447203 0loss/SRqqt4/detection_loss‚2.050698620743222 %loss/total_loss‚39.98379332291462 .loss/FRt1/detection_loss‚3.104169635772705 /loss/FRt2/detection_loss‚2.3073910200024588 /loss/FRt4/assignment_loss‚2.938301347196102 1loss/SRqqt1/assignment_loss‚1.392855716869235 1loss/SRqqt3/detection_loss‚0.8388997120695694 .loss/FBt1/detection_loss‚2.332393386147239 /loss/FBt2/detection_loss‚2.3537337596599874 /loss/FBt3/detection_loss‚0.4320072028785944‰îÁ`¢”ƒ#‚(:‰ ߺëÔá gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“:N—}„#‚(:ò ߺëÔú·Šâ disk.rbd0.in‚0 disk.rbd2.out‚ 2.74609375! disk.nvme0n1.in‚ 6954.98046875 network.recv‚ 3.147617e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 70.9609375( proc.memory.availableMB‚ 1314.2265625 proc.cpu.threads‚17 disk.md0.out‚ 1468.1796875! disk.sda.out‚1506.07080078125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.21676709926625 disk.sda1.out‚0! disk.dm-0.out‚170669.42578125! disk.sdb.out‚1506.07080078125" disk.sdb2.out‚1506.07080078125 disk.md0p1.out‚ 1468.1796875 disk.sda.in‚ 128.6328125) disk./.usagePercent‚76.03582223836257% disk./.usageGB‚2719.4093017578125# disk.nvme0n1.out‚122614.3984375 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1506.07080078125 disk.sdb.in‚ 70.9609375$ memory_percent‚91.97859764099121 network.sent‚ 2.0917376e+07 disk.rbd0.out‚ 2.74609375 disk.rbd2.in‚0 disk.md0.in‚ 199.59375 disk.nvme1n1.in‚ 98.1953125# disk.nvme1n1.out‚48055.02734375 disk.sda1.in‚0 disk.dm-0.in‚ 7053.17578125 disk.md0p1.in‚ 199.59375æÕ†Ý|…#‚(:ñ îºëÔæÚüà disk.dm-0.in‚ 7053.1953125$ memory_percent‚91.98017120361328 disk.md0.out‚ 1469.9609375 disk.nvme1n1.in‚ 98.21484375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 6954.98046875* proc.memory.percent‚18.171024322509766 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375 network.sent‚ 2.1014438e+07 network.recv‚ 3.152184e+06 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625! disk.dm-0.out‚170787.45703125 disk.sdb2.in‚ 70.9609375 cpu‚25.216458430289315 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375& proc.memory.availableMB‚ 1313.96875) disk./.usagePercent‚76.03582383827056 proc.cpu.threads‚17# disk.nvme1n1.out‚48055.13671875 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.74609375 disk.sda1.out‚0# disk.nvme0n1.out‚122732.3203125 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1507.91357421875" disk.sdb2.out‚1507.91357421875% disk./.usageGB‚2719.4093589782715! disk.sda.out‚1507.91357421875 disk.md0p1.out‚ 1469.9609375" disk.sda2.out‚1507.91357421875¤A”»”†#‚(:‰ îºëÔŽ£Ëà gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚128.665' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.16625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±`rJ”‡#‚(:‰ ýºëÔ¯žùÞ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"‹Ødwˆ#‚(:ì ýºëÔ°ºêà network.sent‚ 2.1022861e+07 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6969.38671875$ disk.nvme0n1.out‚122770.92578125 disk.rbd0.out‚ 2.74609375 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1509.79345703125 disk.md0p1.out‚ 1471.76953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 199.59375 disk.md0.out‚ 1471.76953125! disk.sdb.out‚1509.79345703125& proc.memory.availableMB‚ 1313.71875 cpu‚25.216119185094342! disk./.usageGB‚2719.134765625 disk.dm-0.in‚ 7067.609375" disk.sdb2.out‚1509.79345703125! proc.memory.rssMB‚ 2977.140625 disk.sda2.in‚ 128.6328125) disk./.usagePercent‚76.02814609318683$ memory_percent‚91.98169708251953* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.sda2.out‚1509.79345703125 network.recv‚ 3.1566e+06 disk.dm-0.out‚170826.7109375# disk.nvme1n1.out‚48055.78515625 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 98.22265625 disk.sda.in‚ 128.6328125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375ÚŸ¦2‰#- _wandbruntime‚9531  _runtime‚9531–J–ÈŠ# $ _timestamp‚1.788534148117579e+09 _runtime‚9531.653127936 _step‚650 ' lr-AdamW/pg1‚0.00023181423621215444 ' lr-AdamW/pg2‚0.00023181423621215444 trainer/global_step‚7949ÿN˜Í‹#Ç _step‚650 ' lr-AdamW/pg1‚0.00023181423621215444 ' lr-AdamW/pg2‚0.00023181423621215444 trainer/global_step‚7949 $ _timestamp‚1.788534148117579e+09 _runtime‚9531.653127936ŠF Ç2Œ#- _wandbruntime‚9531  _runtime‚9531ÕýùÊü'#öO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201298565 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚656342466823926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚647272796039536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166132720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 367431287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚738584500072960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678508993205248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678509921179648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166132722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48198821 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656341541809562 % _timestamp‚1.7885341481187298e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚669406211952128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669406211952128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9068762809386 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9103709227520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748092452056064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9102799256064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678508993205248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9507975923200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚647272779000176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚9531.654358542 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367404941 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201272221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48198182 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9069670784390 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166132720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚738584476132864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚7949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669406193949184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9103709227520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201272221 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147737057 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99538236 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748092702681088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678509921179648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166132722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48198821 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656341541809562 % _timestamp‚1.7885341481187298e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚669406211952128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669406211952128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9068762809386 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9103709227520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748092452056064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9102799256064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678508993205248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9507975923200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚647272779000176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚9531.654358542 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367404941 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201272221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48198182 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9069670784390 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166132720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚738584476132864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚7949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669406193949184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9103709227520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201272221 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147737057 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99538236 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748092702681088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9104830701056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.f SáÆk reed‚ 367449932 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367476304 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201297052 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678591733751296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9509364092416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166152880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166152880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48204576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚9532.644034599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚738674701837312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147754877 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656422601691956 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669487813035008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201297052 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99550299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9104830701056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885341491075015e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201323422 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669487831037952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367476304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656421676677384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9103920716288 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147754222 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚647351813634080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367449932 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9104830701056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367449932 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367476304 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201297052 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678591733751296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9509364092416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166152880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166152880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48204576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚9532.644034599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚738674701837312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147754877 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656422601691956 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669487813035008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201297052 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99550299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9104830701056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885341491075015e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201323422 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669487831037952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367476304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656421676677384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9103920716288 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147754222 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚647351813634080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367449932 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚752662425511936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752662678234112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660348298791870 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202517123 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202543467 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9159796652544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202517123 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9566892938752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167140720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682646034921472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369657843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚743095785295360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651223675766736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167140720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9158886681088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369684189 VHDeviceStatsMonitor.on_train_batch_É<éstart/inactive_split_bytes.all.current‚ 252722176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9124640064494 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48489444 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 # _timestamp‚1.78853420289776e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682646962895872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673487148240384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682646962895872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7999 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167140722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9566664156672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660349223806234 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651223658727376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚655 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚673487148240384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100141486 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48490091 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚743095761355264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9158886681088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148631577 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673487166243328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚9586.433358917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100141484ˆº(¢#ûO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682646962895872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673487148240384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682646962895872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7999 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167140722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9566664156672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660349223806234 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651223658727376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚655 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚673487148240384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100141486 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48490091 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚743095761355264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9158886681088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148631577 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673487166243328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚9586.433358917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100141484 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9125548039498 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚682646034921472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚752662425511936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752662678234112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660348298791870 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202517123 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202543467 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9159796652544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202517123 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9566892938752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167140720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682646034921472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369657843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚743095785295360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651223675766736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167140720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9158886681088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369684189 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9124640064494 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48489444 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 # _timestamp‚1.78853420289776e+09¦~ÖF2£#- _wandbruntime‚9587  _runtime‚9587ˆÐ“w'¤#‚N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673568785329152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚752753824248320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167160880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚743186010999808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚682729703455232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651302693361280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673568767326208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202568324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9125757337772 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9160008141312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743185987059712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660428433659692 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100153549 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 trainer/global_step‚7999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚682728775467520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167160880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202541954 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9160918126080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167160882 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚673568767326208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48495149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651302676321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 % _timestamp‚1.7885342038862789e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48495781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369729206 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148649332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9568061763072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167160882 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9126665312984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100153551 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752754072762880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369702834 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660429358674264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369729206 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673568785329152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202568324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚682729703455232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9160008141312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148648698 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369702834 _runtime‚9587.422472815ZÉ1B¢¥#‡N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚682728775467520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167160880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202541954 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9160918126080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167160882 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚673568767326208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48495149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651302676321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 % _timestamp‚1.7885342038862789e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48495781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369729206 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148649332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9568061763072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167160882 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9126665312984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100153551 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752754072762880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369702834 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660429358674264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369729206 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673568785329152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202568324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚682729703455232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9160008141312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148648698 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369702834 _runtime‚9587.422472815 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673568785329152 XDDeviceStatî'qk sMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚752753824248320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167160880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚743186010999808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚682729703455232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651302693361280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673568767326208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202568324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9125757337772 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9160008141312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743185987059712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660428433659692 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100153549 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 trainer/global_step‚7999:ç}™2¦#- _wandbruntime‚9587  _runtime‚9587'Q6­ª”°#‚(:‰ õ»ëԣĭà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ H©2±#- _wandbruntime‚9646  _runtime‚9646 _tɲ#à % _timestamp‚1.7885342625625641e+09 _runtime‚9646.098563758 _step‚658 ' lr-AdamW/pg1‚0.00022204202429720068 ' lr-AdamW/pg2‚0.00022204202429720068 trainer/global_step‚8049YŒšÎ³#È _runtime‚9646.098563758 _step‚658 ' lr-AdamW/pg1‚0.00022204202429720068 ' lr-AdamW/pg2‚0.00022204202429720068 trainer/global_step‚8049 % _timestamp‚1.7885342625625641e+09’(9ãv2´#- _wandbruntime‚9646  _runtime‚9646–”¤þ'µ#øO YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100744736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371937091 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203762025 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149525301 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚757232564295680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚747607070517760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686783076637696 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168148722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203762025 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885342625645752e+09 _runtime‚9646.100739928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664355055774178 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9181425294606 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚655174538454576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9180517319602 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9214974106112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686784004612096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203788369 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168148722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747607046577664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚659 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677568120534528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168148720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9625493777920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686783076637696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677568102531584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9625254510080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885342625645752e+09 _runtime‚9646.100739928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664355055774178 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9181425294606 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚655174538454576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9180517319602 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9214974106112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686784004612096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203788369 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168148722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747607046577664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚659 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677568120534528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168148720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9625493777920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686783076637696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677568102531584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9625254510080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371937091 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203762025“ŒE.2·#- _wandbruntime‚9647  _runtime‚9647ï\å'¸#‚N FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9647.331598426 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203813226 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168168882 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677649739620352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9626646053376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚664436115656572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48786290 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371955736 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100756799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168168882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655253573088480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686865817183744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚757323685372416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664435190642000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203786856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9217005551104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168168880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168168880 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100756801 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655253556049120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9181634592880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149543089 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677649721617408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149543751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677649739620352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371982108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚747697296222208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48786950 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371955736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168168882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655253573088480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686865817183744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚757323685372416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664435190642000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203786856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9217005551104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168168880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168168880 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100756801 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655253556049120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9181634592880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149543089 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677649721617408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149543751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677649739620352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371982108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚747697296222208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48786950 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371955736 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9647.331598426 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203813226 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168168882 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677649739620352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9626646053376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚664436115656572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0”±VP2º#- _wandbruntime‚9647  _runtime‚9647í¸ `»#Ú .loss/FRt2/detection_loss‚2.437713876019905 1loss/SRqqt4/detection_loss‚2.4181241146449386 %loss/total_loss‚40.25138324787527 0loss/FBt3/detection_loss‚0.37355372151796473 0loss/FRt2/assignment_loss‚3.3284079233805337 0loss/FRt4/assignment_loss‚2.7573933516229903 2loss/SRqqt4/assignment_loss‚1.2267239430341228 1loss/FBt4/assignment_loss‚0.18766272997133626 2loss/SRqqt2/assignment_loss‚1.3116243271423238 1loss/FBt2/assignment_loss‚0.18417194298817682 .loss/FRt1/detection_loss‚2.674889421463013 1loss/SRqqt1/detection_loss‚2.4034008528247024 0loss/SRqqt3/detection_loss‚0.755017388546676 % _timestamp‚1.7885342637974896e+09 /loss/FRt3/assignment_loss‚0.854242049730741 2loss/SRqqt1/assignment_loss‚1.0916625107779647 1loss/SRqqt2/detection_loss‚2.6484595047576085 /loss/FBt1/detection_loss‚2.4219882951842413 trainer/global_step‚8049 1loss/FBt1/assignment_loss‚0.43671169945506944 1loss/FBt3/assignment_loss‚0.11585127017834118 /loss/FBt4/detection_loss‚2.4640810191631317 /loss/FRt3/detection_loss‚1.2982835602015257 /loss/FRt4/detection_loss‚3.1410407051444054 /loss/FBt2/detection_loss‚1.9758029599984488 epoch‚6 _runtime‚9647.333099863 _step‚661 0loss/FRt1/assignment_loss‚3.0617614845434824 2loss/SRqqt3/assignment_loss‚0.6828145955836302%ÎŒ’e¼#ß 1loss/FBt3/assignment_loss‚0.11585127017834118 /loss/FBt4/detection_loss‚2.4640810191631317 /loss/FRt3/detection_loss‚1.2982835602015257 /loss/FRt4/detection_loss‚3.1410407051444054 /loss/FBt2/detection_loss‚1.9758029599984488 epoch‚6 _runtime‚9647.333099863 _step‚661 0loss/FRt1/assignment_loss‚3.0617614845434824 2loss/SRqqt3/assignment_loss‚0.6828145955836302 .loss/FRt2/detection_loss‚2.437713876019905 1loss/SRqqt4/detection_loss‚2.4181241146449386 %loss/total_loss‚40.25138324787527 0loss/FBt3/detection_loss‚0.37355372151796473 0loss/FRt2/assignment_loss‚3.3284079233805337 0loss/FRt4/assignment_loss‚2.7573933516229903 2loss/SRqqt4/assignment_loss‚1.2267239430341228 1loss/FBt4/assignment_loss‚0.18766272997133626 2loss/SRqqt2/assignment_loss‚1.3116243271423238 1loss/FBt2/assignment_loss‚0.18417194298817682 .loss/FRt1/detection_loss‚2.674889421463013 1loss/SRqqt1/detection_loss‚2.4034008528247024 0loss/SRqqt3/detection_loss‚0.755017388546676 % _timestamp‚1.7885342637974896e+09 /loss/FRt3/assignment_loss‚0.854242049730741 2loss/SRqqt1/assignment_loss‚1.0916625107779647 1loss/SRqqt2/detection_loss‚2.6484595047576085 /loss/FBt1/detection_loss‚2.4219882951842413 trainer/global_step‚8049 1loss/FBt1/assignment_loss‚0.43671169945506944•ûùU”½#‚(:‰ „¼ëÔƒ¼¥à gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04èm€¾#‚(:õ „¼ëÔ¯¥âà# disk.nvme1n1.out‚48897.48828125! disk.nvme0n1.in‚ 6969.41796875 disk.md0.out‚ 1485.98828125 disk.sdb.out‚1524.5244140625 disk.sda.in‚ 128.6328125 disk.sda.out‚1524.5244140625 proc.cpu.threads‚17 network.sent‚ 2.1290069e+07 disk.nvme1n1.in‚ 98.26171875 disk.sdb1.out‚0 disk.rbd2.out‚ 2.8994140625! disk.sda2.out‚1524.5244140625! disk.dm-0.out‚172213.71484375" memory_percent‚91.983962059021 cpu‚25.213493947747722) disk./.usagePercent‚76.06112009003174% disk./.usageGB‚2720.3140754699707 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0 disk.rbd2.in‚0 network.recv‚ 3.196317e+06# disk.nvme0n1.out‚123316.2265625 disk.md0.in‚ 199.59375 disk.md0p1.out‚ 1485.98828125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375! disk.sdb2.out‚1524.5244140625 disk.rbd0.out‚ 2.92236328125 disk.dm-0.in‚ 7067.6796875 disk.sdb.in‚ 70.9609375) proc.memory.availableMB‚ 1313.34765625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.rbd0.in‚0Áñ0<”¿#‚(:‰ “¼ëÔ¸òØà gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÙ©÷}À#‚(:ò “¼ëÔÓ€½á( proc.memory.availableMB‚ 1313.1015625* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06461066260594$ disk.nvme0n1.out‚123316.87109375" disk.sda2.out‚1526.22509765625! disk.sdb.out‚1526.22509765625 disk.nvme1n1.in‚ 98.26171875# disk.nvme1n1.out‚49058.31640625$ memory_percent‚91.98546409606934" disk.sdb2.out‚1526.22509765625 disk.md0p1.out‚ 1487.640625 disk.sda1.out‚0% disk./.usageGB‚2720.4389152526855! disk.nvme0n1.in‚ 6969.41796875 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 cpu‚25.213211490174615 network.sent‚ 2.1384549e+07 disk.rbd2.out‚ 2.8994140625 disk.dm-0.in‚ 7067.6796875 disk.dm-0.out‚ 172375.1875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd0.out‚ 2.9619140625 disk.rbd2.in‚0 disk.sda.in‚ 128.6328125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.md0.out‚ 1487.640625 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1526.22509765625! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.201135e+06 disk.sda2.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0~‘9’Á#‚(:‡ ¢¼ëÔŒž½à gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉûÜQÂ#‚(:ô ¢¼ëÔóÝûà! disk.sda2.out‚1527.7685546875' proc.memory.availableMB‚ 1313.109375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.out‚ 1489.12109375! disk.dm-0.out‚172537.23046875 disk.sdb2.in‚ 70.9609375% disk./.usageGB‚2720.5644874572754! disk.nvme0n1.in‚ 6969.41796875 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 2.9619140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2977.140625 disk.sdb.in‚ 70.9609375) disk./.usagePercent‚76.06812171400233 disk.sda.in‚ 128.6328125 disk.sda.out‚1527.7685546875 disk.md0p1.out‚ 1489.12109375 disk.rbd2.out‚ 2.96240234375 disk.sdb.out‚1527.7685546875$ memory_percent‚91.98541641235352 disk.sdb1.in‚0 network.sent‚ 2.1392433e+07$ disk.nvme0n1.out‚123316.87109375* proc.memory.percent‚18.171024322509766! disk.nvme1n1.out‚ 49220.359375 disk.rbd0.in‚0! disk.sdb2.out‚1527.7685546875 cpu‚25.212885041097522 disk.dm-0.in‚ 7067.6796875 network.recv‚ 3.205058e+06 disk.nvme1n1.in‚ 98.26171875 disk.sda2.in‚ 128.6328125òK¯ƒ“Ã#‚(:ˆ ±¼ëԚׂá gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸK´yÄ#‚(:î ±¼ëÔíþÃá disk.sda.in‚ 128.6328125 disk.sda.out‚1529.599609375! disk.nvme0n1.in‚ 6969.41796875 disk.md0.in‚ 199.59375 proc.cpu.threads‚17! disk.dm-0.out‚172806.73828125 disk.sdb.out‚1529.599609375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0! disk.nvme0n1.out‚ 123317.46875 disk.md0.out‚ 1490.8828125' proc.memory.availableMB‚ 1313.359375 cpu‚25.212579435069546 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1490.8828125 disk.sda2.out‚1529.599609375* proc.memory.percent‚18.171024322509766 network.sent‚ 2.1396503e+07) disk./.usagePercent‚76.06810571492251% disk./.usageGB‚2720.5639152526855 disk.sdb2.in‚ 70.9609375 network.recv‚ 3.209811e+06 disk.nvme1n1.in‚ 98.26171875 disk.rbd0.out‚ 3.02490234375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.96240234375 disk.sdb1.out‚0! proc.memory.rssMB‚ 2977.140625 disk.dm-0.in‚ 7067.6796875# disk.nvme1n1.out‚49489.26953125 disk.sda1.out‚0 disk.sdb2.out‚1529.599609375 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125$ memory_percent‚91.98389053344727RËyö2Å#- _wandbruntime‚9707  _runtime‚97073̇±ÉÆ#à ' lr-AdamW/pg1‚0.00021720038031789192 ' lr-AdamW/pg2‚0.00021720038031789192 trainer/global_step‚8099 % _timestamp‚1.7885343236178172e+09 _runtime‚9707.153567145 _step‚662óä ]ÎÇ#È ' lr-AdamW/pg1‚0.00021720038031789192 ' lr-AdamW/pg2‚0.00021720038031789192 trainer/global_step‚8099 % _timestamp‚1.7885343236178172e+09 _runtime‚9707.153567145 _step‚662–Ð?5ˆ2È#- _wandbruntime‚9707  _runtime‚9707ûä3Mÿ É#øO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9237302549714 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681649074825728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374163647 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169156720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169156722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9683838975488 _step‚663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205006927 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690921046328320 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9271061531136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690920118353920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9683605999104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚8099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9271061531136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚761802194715648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9236394574710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9271971502592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49070966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169156722 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49071613 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚761801937799168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205006927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690920118353920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690921046328320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681649056822784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668361812756486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659125418181776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚9707.154505527 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150419599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681649056822784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9271971502592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9271061531136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690920118353920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9683605999104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚8099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9271061531136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚761802194715648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9236394574710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9271971502592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49070966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169156722 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49071613 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚761801937799168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205006927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690920118353920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690921046328320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681649056822784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668361812756486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659125418181776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚9707.154505527 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150419599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681649056822784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9271971502592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150436799 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9273092976128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681730693911552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101360049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169176882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374208638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101360051 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374235010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49077400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9272182991360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668441947624308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659204435776320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788534324786898e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9272182991360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169176882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49076750 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374235010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712¤û`b¢Í#†N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169176882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374208638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101360051 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374235010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49077400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9272182991360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668441947624308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659204435776320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788534324786898e+09 UCDevic’É>äjeStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9272182991360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169176882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49076750 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374235010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205031758 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205058128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150437451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9684774126080 trainer/global_step‚8099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚761893584339456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150436799 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9273092976128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681730693911552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101360049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570˜ˆ¼'ñ2Î#- _wandbruntime‚9708  _runtime‚9708çØeÏ#ß _runtime‚9708.324965867 3loss/FBt1/assignment_loss‚0.0019988451054814504 .loss/FRt1/detection_loss‚2.683595138065743 1loss/SRqqt4/detection_loss‚2.2236147197810086 epoch‚6 trainer/global_step‚8099 2loss/FBt2/assignment_loss‚0.042390202172100544 .loss/FRt2/detection_loss‚2.328838342295577 /loss/FRt3/detection_loss‚1.3957686304638628 1loss/SRqqt1/detection_loss‚2.5887453811509267 /loss/FBt2/detection_loss‚2.2150611907243727 0loss/SRqqt2/detection_loss‚2.473205524521905 1loss/SRqqt3/detection_loss‚0.8464614876679012 0loss/FRt1/assignment_loss‚3.2198362531084004 2loss/SRqqt2/assignment_loss‚1.7172924022416811 &loss/total_loss‚41.965042215254215 1loss/FBt4/assignment_loss‚0.25552740726440754 /loss/FBt1/detection_loss‚2.2670612156391146 0loss/FBt3/detection_loss‚0.40197611995209315 0loss/FRt2/assignment_loss‚3.7125227553503852 2loss/SRqqt3/assignment_loss‚0.8625280974876313 /loss/FBt4/detection_loss‚2.4354445891720906 _step‚665 0loss/FRt3/assignment_loss‚1.5131911113858223 0loss/FRt4/assignment_loss‚3.3741963946301006 2loss/SRqqt1/assignment_loss‚1.6063615674419063 2loss/SRqqt4/assignment_loss‚1.4499883516268297 1loss/FBt3/assignment_loss‚0.06020787334221455 /loss/FRt4/detection_loss‚2.2892286146626524 % _timestamp‚1.7885343247896707e+09]î]ÆjÐ#ä 0loss/FRt2/assignment_loss‚3.7125227553503852 2loss/SRqqt3/assignment_loss‚0.8625280974876313 /loss/FBt4/detection_loss‚2.4354445891720906 _step‚665 0loss/FRt3/assignment_loss‚1.5131911113858223 0loss/FRt4/assignment_loss‚3.3741963946301006 2loss/SRqqt1/assignment_loss‚1.6063615674419063 2loss/SRqqt4/assignment_loss‚1.4499883516268297 1loss/FBt3/assignment_loss‚0.06020787334221455 /loss/FRt4/detection_loss‚2.2892286146626524 % _timestamp‚1.7885343247896707e+09 _runtime‚9708.324965867 3loss/FBt1/assignment_loss‚0.0019988451054814504 .loss/FRt1/detection_loss‚2.683595138065743 1loss/SRqqt4/detection_loss‚2.2236147197810086 epoch‚6 trainer/global_step‚8099 2loss/FBt2/assignment_loss‚0.042390202172100544 .loss/FRt2/detection_loss‚2.328838342295577 /loss/FRt3/detection_loss‚1.3957686304638628 1loss/SRqqt1/detection_loss‚2.5887453811509267 /loss/FBt2/detection_loss‚2.2150611907243727 0loss/SRqqt2/detection_loss‚2.473205524521905 1loss/SRqqt3/detection_loss‚0.8464614876679012 0loss/FRt1/assignment_loss‚3.2198362531084004 2loss/SRqqt2/assignment_loss‚1.7172924022416811 &loss/total_loss‚41.965042215254215 1loss/FBt4/assignment_loss‚0.25552740726440754 /loss/FBt1/detection_loss‚2.2670612156391146 0loss/FBt3/detection_loss‚0.40197611995209315™ÝMpÛ”Ñ#‚(:‰ À¼ëÔ܃Òà gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'„È}‚Ò#‚(:÷ À¼ëÔ‡Š¯â network.sent‚ 2.1415551e+07 disk.md0.in‚ 199.59375" disk.sdb2.out‚1530.97998046875 disk.md0p1.out‚ 1492.20703125 disk.sda1.in‚0 disk.dm-0.in‚ 7067.6796875 disk.sdb.in‚ 70.9609375 proc.cpu.threads‚17 disk.sda.in‚ 128.6328125 disk.rbd0.out‚ 3.02490234375$ memory_percent‚91.98403358459473! proc.memory.rssMB‚ 2977.140625 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6969.41796875 cpu‚25.212248226766448 disk.nvme1n1.in‚ 98.26171875 disk.sda1.out‚0! disk.dm-0.out‚173036.86328125# disk.nvme1n1.out‚49719.39453125 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06810731483048% disk./.usageGB‚2720.5639724731445 disk.rbd2.in‚0 disk.sda2.in‚ 128.6328125" disk.sda2.out‚1530.97998046875! disk.sdb.out‚1530.97998046875( proc.memory.availableMB‚ 1313.3359375 network.recv‚ 3.214254e+06! disk.sda.out‚1530.97998046875 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.out‚ 123317.46875 disk.md0.out‚ 1492.20703125IÛ¿±”Ó#‚(:‰ ϼëÔŒÿ¨à gpu.0.gpu‚25.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.067' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.76675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾=øevÔ#‚(:ë ϼëÔèÏúà disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 disk.sda2.in‚ 128.6328125" disk.sdb2.out‚1532.69482421875 disk.nvme1n1.in‚ 98.26171875 disk.md0p1.out‚ 1493.8515625 disk.rbd2.out‚ 3.025390625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb.in‚ 70.9609375% disk./.usageGB‚2720.2891387939453$ disk.nvme0n1.out‚123317.62890625" disk.sda2.out‚1532.69482421875! disk.nvme0n1.in‚ 6969.83203125 disk.md0.out‚ 1493.8515625! disk.sda.out‚1532.69482421875 disk.dm-0.in‚ 7068.09375! proc.memory.rssMB‚ 2977.140625! disk.sdb.out‚1532.69482421875 network.recv‚ 3.219167e+06 disk.rbd2.in‚0) proc.memory.availableMB‚ 1313.21484375* proc.memory.percent‚18.171024322509766 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0) disk./.usagePercent‚76.06042285013325 disk.dm-0.out‚173159# disk.nvme1n1.out‚49841.37109375 cpu‚25.21190337878276 proc.cpu.threads‚17 network.sent‚ 2.1512227e+07 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.59375$ memory_percent‚91.98477268218994^XYì’Õ#‚(:‡ Þ¼ëÔîÛÕà gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qWÍ{Ö#‚(:ð Þ¼ëÔžá disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1534.43798828125# disk.nvme1n1.out‚49997.43359375 disk.md0p1.in‚ 199.59375 network.sent‚ 2.1515626e+07 disk.sda1.out‚0 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.in‚ 6969.83203125! disk.sdb.out‚1534.43798828125 disk.nvme1n1.in‚ 98.453125 disk.md0p1.out‚ 1495.52734375 disk.rbd0.out‚ 3.087890625) disk./.usagePercent‚76.05933405942135$ disk./.usageGB‚2720.250198364258 disk.dm-0.out‚ 173315.0625$ memory_percent‚91.98474884033203 cpu‚25.211577440641896 network.recv‚ 3.222699e+06 disk.rbd0.in‚0* proc.memory.percent‚18.171024322509766 disk.sda2.in‚ 128.6328125 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2977.140625" disk.sda2.out‚1534.43798828125 disk.md0.out‚ 1495.52734375 disk.dm-0.in‚ 7068.28515625 disk.sdb.in‚ 70.9609375! disk.sda.out‚1534.43798828125 disk.sda1.in‚0$ disk.nvme0n1.out‚123317.62890625& proc.memory.availableMB‚ 1313.21875 proc.cpu.threads‚17‹\ÿ’×#‚(:‡ í¼ëÔ»¸ˆß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-—w~Ø#‚(:ó í¼ëÔ¼¿Âá disk.nvme1n1.in‚ 100.4296875# disk.nvme1n1.out‚50164.95703125 disk.rbd2.in‚0! disk.nvme0n1.in‚ 6969.83203125$ memory_percent‚91.98629856109619) proc.memory.availableMB‚ 1312.96484375 network.recv‚ 3.22736e+06 disk.sdb.in‚ 70.9609375 disk.md0.in‚ 199.59375* proc.memory.percent‚18.171024322509766! disk.sdb.out‚1535.82080078125$ disk.nvme0n1.out‚123317.73046875" disk.sda2.out‚1535.82080078125 cpu‚25.21130204085646) disk./.usagePercent‚76.06112670298472" disk.sdb2.out‚1535.82080078125! disk.sda.out‚1535.82080078125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0$ disk./.usageGB‚2720.314311981201 disk.dm-0.in‚ 7070.26171875 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.rbd2.out‚ 3.025390625 disk.md0.out‚ 1496.84765625 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.1524776e+07 disk.md0p1.out‚ 1496.84765625 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 disk.dm-0.out‚ 173482.6875N•Ó2Ù#- _wandbruntime‚9768  _runtime‚9768zD‚µÇÚ#Á ' lr-AdamW/pg1‚0.00021238972506173697 ' lr-AdamW/pg2‚0.00021238972506173697 trainer/global_step‚8149 % _timestamp‚1.7885343852473936e+09 _runtime‚ 9768.7830327 _step‚666Ï©-¸ÌÛ#Æ % _timestamp‚1.7885343852473936e+09 _runtime‚ 9768.7830327 _step‚666 ' lr-AdamW/pg1‚0.00021238972506173697 ' lr-AdamW/pg2‚0.00021238972506173697 trainer/global_step‚8149šÒš> 2Ü#- _wandbruntime‚9768  _runtime‚9768û-ë´ý'Ý#÷O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695058088044544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788534385248609e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚695058088044544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9327148956160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170164722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756629617022464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672369494753158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206278173 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233847808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚766371990595072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9742373572608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695057160070144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49363138 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756629640962560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672368569738794 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170164722 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚766372224442880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚8149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101951236 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685730029116928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206251829 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚9768.784263136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9327148956160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663076297908976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376416549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695057160070144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685730029116928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663076314948336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209907712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206251829 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚9768.784263136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9327148956160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663076297908976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376416549 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695057160070144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685730029116928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663076314948336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209907712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695058088044544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788534385248609e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚695058088044544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9327148956160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170164722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756629617022464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672369494753158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206278173 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233847808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚766371990595072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9742373572608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695057160070144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49363138 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756629640962560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0›væÇT2ß#- _wandbruntime‚9769  _runtime‚9769ÚPžv'à#‚N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170184882 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376487912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206303030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9743762043904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376461540 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170184882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376461540 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206276660 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9328270416384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚766463628710912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756719842726912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170184880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206303030 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663155315503520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685811630199808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚9769.751887668 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9328270416384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151331582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9328270416384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151331582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376487912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206303030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9743762043904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376461540 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170184882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376461540 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206276660 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9328270416384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚766463628710912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756719842726912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170184880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206303030 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663155315503520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685811630199808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚9769.751887668 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152208186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378669451 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207523075 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699194201786368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378695797 _step‚671 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171172722 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689810983408128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9384146352640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9383236381184 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152208833 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171172720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9348149084926 trainer/global_step‚8199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207496731 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171172722 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102554484 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761140926184960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚676376251735466 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699194201786368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689810965405184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9384146352640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712Óák½(ð#üO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49653702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207496731 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171172722 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102554484 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761140926184960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚676376251735466 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699194201786368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689810965405184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9384146352640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699195129760768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667027177636176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699195129760768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207523075 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667027194675536 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207496731 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102554486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9801120615936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171172720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9822.437118849 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚770942269291520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9383236381184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152208186 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378669451 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207523075 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699194201786368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378695797 _step‚671 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171172722 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689810983408128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9384146352640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9383236381184 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152208833 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171172720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9348149084926 trainer/global_step‚8199 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152225951 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152226604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771033668450816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761231151889408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚699277870320128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689892584491008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378740814 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171192880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9385267826176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171192882 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771033417839104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171192880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667106195230720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚9823.680508549 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9349266358204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 $ _timestamp‚1.788534440142767e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378714442 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9802289889792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171192882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102566551 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350174333416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689892602493952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚676455461588924 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9802516561408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207521562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689892584491008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49660053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378714442 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689892602493952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368ÇoÜ€¯ó#†N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49660053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378714442 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689892602493952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207547932 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚699277870320128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9384357841408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚761231127949312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚699276942332416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152225951 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152226604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771033668450816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761231151889408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚699277870320128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689892584491008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378740814 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171192880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9385267826176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171192882 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771033417839104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171192880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667106195230720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚9823.680508549 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9349266358204 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 $ _timestamp‚1.788534440142767e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378714442 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9802289889792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171192882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102566551 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350174333416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689892602493952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚676455461588924 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9802516561408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207521562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689892584491008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670978057363376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693891937699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9440233777664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚765652211407360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693891919696384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172180722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚775511855892480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670978074402736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172180720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚775512102323200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703331243502592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208741633 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 $ _timestamp‚1.788534492714697e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49945625 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9440233777664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9439323806208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208767977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172180720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153103361 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153102747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103157736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ó·Íá„$üO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚775512102323200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703331243502592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208741633 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 $ _timestamp‚1.788534492714697e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49945625 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9440233777664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9439323806208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208767977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172180720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153103361 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153102747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103157736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚693891937699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9859890915840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380922353 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 trainer/global_step‚8249 _runtime‚9876.250483537 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9439323806208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172180722 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚693891919696384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680383008717774 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703331243502592 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103157734 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9404934315038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703332171476992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208741633 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚680382083703410 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9404026340034 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670978057363376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693891937699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9440233777664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚765652211407360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693891919696384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172180722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚775511855892480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670978074402736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172180720£“y`¸2…$- _wandbruntime‚9877  _runtime‚9877ØûkG'†$‚N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9405143613312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚676 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208766464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚775603501416960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380967344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9440445266432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9440445266432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 trainer/global_step‚8249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9877.567811347 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9441355251200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703413984048640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693973556785152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103169801 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380993716 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153121173 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680462218571232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208766464 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208792834 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703414912036352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703414912036352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765742437111808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885344940292554e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9861064305152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚671057074957920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680463143585804 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172200882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153120560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49951372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693973556785152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693973538782208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775603250805248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚671057091997280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380993716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172200880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406051588524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208792834 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172200880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9860837633536 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703413984048640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693973556785152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103169801 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380993716 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153121173 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680462218571232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208766464 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208792834 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703414912036352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703414912036352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765742437111808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885344940292554e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9861064305152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚671057074957920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680463143585804 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172200882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153120560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49951372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693973556785152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693973538782208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775603250805248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚671057091997280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380993716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172200880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406051588524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208792834 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172200880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9860837633536 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9440445266432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9440445266432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 trainer/global_step‚8249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9877.567811347 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9441355251200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¤,Æ2ˆ$- _wandbruntime‚9877  _runtime‚9877þY‰$Ó 1loss/SRqqt3/detection_loss‚0.8239876109799114 2loss/SRqqt3/assignment_loss‚0.8777745424111912 _step‚677 /loss/FRt1/detection_loss‚2.9151163617769877 -loss/FBt4/detection_loss‚2.17752468585968 0loss/FRt1/assignment_loss‚3.1369337598482767 0loss/FBt2/assignment_loss‚0.2055723284710049 0loss/FBt4/assignment_loss‚0.0736350942888728 /loss/FBt3/detection_loss‚0.4705715680029243 %loss/total_loss‚40.17300400106914 0loss/FBt3/assignment_loss‚0.2040525353463848 .loss/FRt2/detection_loss‚2.266124307643622 1loss/SRqqt2/detection_loss‚2.1293512126978706 1loss/SRqqt4/detection_loss‚2.1462847696883336 epoch‚6 $ _timestamp‚1.788534494033832e+09 0loss/FRt2/assignment_loss‚3.3298300536055314 1loss/SRqqt1/detection_loss‚2.5442805928843364 .loss/FBt1/detection_loss‚2.321713374211238 .loss/FBt2/detection_loss‚1.806651886020388 2loss/SRqqt1/assignment_loss‚1.5148798514689719 2loss/SRqqt2/assignment_loss‚1.3305096765432287 2loss/SRqqt4/assignment_loss‚1.3846614883680428 /loss/FRt3/detection_loss‚1.5655078682738046 .loss/FRt4/detection_loss‚2.573674973845482 trainer/global_step‚8249 _runtime‚9877.569307274 /loss/FRt3/assignment_loss‚1.168678449591001 0loss/FRt4/assignment_loss‚3.1627728819847105 2loss/FBt1/assignment_loss‚0.042914127257343074\À ›^Š$Ø 1loss/SRqqt3/detection_loss‚0.8239876109799114 2loss/SRqqt3/assignment_loss‚0.8777745424111912 _step‚677 /loss/FRt1/detection_loss‚2.9151163617769877 -loss/FBt4/detection_loss‚2.17752468585968 0loss/FRt1/assignment_loss‚3.1369337598482767 0loss/FBt2/assignment_loss‚0.2055723284710049 0loss/FBt4/assignment_loss‚0.0736350942888728 /loss/FBt3/detection_loss‚0.4705715680029243 %loss/total_loss‚40.17300400106914 0loss/FBt3/assignment_loss‚0.2040525353463848 .loss/FRt2/detection_loss‚2.266124307643622 1loss/SRqqt2/detection_loss‚2.1293512126978706 1loss/SRqqt4/detection_loss‚2.1462847696883336 epoch‚6 $ _timestamp‚1.788534494033832e+09 0loss/FRt2/assignment_loss‚3.3298300536055314 1loss/SRqqt1/detection_loss‚2.5442805928843364 .loss/FBt1/detection_loss‚2.321713374211238 .loss/FBt2/detection_loss‚1.806651886020388 2loss/SRqqt1/assignment_loss‚1.5148798514689719 2loss/SRqqt2/assignment_loss‚1.3305096765432287 2loss/SRqqt4/assignment_loss‚1.3846614883680428 /loss/FRt3/detection_loss‚1.5655078682738046 .loss/FRt4/detection_loss‚2.573674973845482 trainer/global_step‚8249 _runtime‚9877.569307274 /loss/FRt3/assignment_loss‚1.168678449591001 0loss/FRt4/assignment_loss‚3.1627728819847105 2loss/FBt1/assignment_loss‚0.042914127257343074¥ÁÆ ¨’‹$‚(:‡ å½ëԬОà gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Ÿ;ÍxŒ$‚(:í å½ëÔ¢ÍÊá disk.md0p1.out‚ 1509.28125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1548.74755859375 network.recv‚ 3.263763e+06 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 disk.rbd2.out‚ 3.025390625 network.sent‚ 2.1785915e+07 disk.dm-0.in‚ 7070.76171875$ memory_percent‚91.98975563049316 proc.cpu.threads‚17" disk.sda2.out‚1548.74755859375# disk.nvme1n1.out‚51132.02734375( proc.memory.availableMB‚ 1312.3984375 cpu‚25.20902030166212 disk.sdb1.out‚0 disk.rbd2.in‚0$ disk./.usageGB‚2719.564712524414 disk.nvme0n1.out‚ 123457.9375 disk.md0.out‚ 1509.28125! disk.dm-0.out‚174589.96484375 disk.sdb.in‚ 70.9609375! disk.sda.out‚1548.74755859375 disk.nvme1n1.in‚ 100.4296875 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1548.74755859375* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.04016758844121 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0! disk.nvme0n1.in‚ 6970.33203125 disk.md0.in‚ 199.59375û>•ä“$‚(:ˆ ô½ëÔÃéÊà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ/etrŽ$‚(:ç ô½ëÔ£€á disk.dm-0.in‚ 7070.76171875 network.recv‚ 3.268429e+06 disk.sdb.in‚ 70.9609375 disk.sda.out‚ 1550.57421875# disk.nvme1n1.out‚51244.97265625 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625( disk./.usagePercent‚76.0401532959299 disk.sdb1.in‚0( proc.memory.availableMB‚ 1312.6328125 cpu‚25.208732169859076 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766 network.sent‚ 2.1880234e+07 disk.sda.in‚ 128.6328125 disk.sda1.in‚0 disk.md0.in‚ 199.59375 disk.md0.out‚ 1511.046875 disk.sdb.out‚ 1550.57421875 disk.nvme1n1.in‚ 100.4296875 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625 disk.md0p1.out‚ 1511.046875 disk.sda2.out‚ 1550.57421875! disk.nvme0n1.in‚ 6970.33203125$ disk.nvme0n1.out‚123462.44140625 disk.dm-0.out‚174707.4140625$ memory_percent‚91.98832511901855 proc.cpu.threads‚17% disk./.usageGB‚2719.5642013549805 disk.sdb2.out‚ 1550.57421875©×u“$‚(:ˆ ƒ¾ëÔ©ÊØà gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚147.606' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.9015 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶„has$‚(:è ƒ¾ëÔ¤ï¢ä disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.sda1.in‚0 disk.md0.out‚ 1512.46875) disk./.usagePercent‚76.04363224250612 disk.rbd0.in‚0# disk.nvme0n1.out‚123514.5546875 disk.dm-0.out‚174872.9921875 disk.sdb.out‚1552.0556640625 disk.nvme1n1.out‚ 51358.4375 disk.md0p1.out‚ 1512.46875* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375% disk./.usageGB‚2719.6886253356934 disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 proc.cpu.threads‚17 disk.sda.in‚ 128.6328125 network.sent‚ 2.1885632e+07 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.in‚ 6970.33203125! disk.sda2.out‚1552.0556640625 disk.dm-0.in‚ 7070.76171875 network.recv‚ 3.272618e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sdb2.out‚1552.0556640625 disk.sda.out‚1552.0556640625 disk.nvme1n1.in‚ 100.4296875# memory_percent‚91.9853687286377( proc.memory.availableMB‚ 1313.1171875! proc.memory.rssMB‚ 2977.140625 cpu‚25.208425666436106º?‘’‘$‚(:‡ ’¾ëÔšòÙÞ gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äç {’$‚(:ð ’¾ëÔ‘„¢à disk.dm-0.in‚ 7070.76171875 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.98613166809082 disk.rbd2.out‚ 3.025390625 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1553.56298828125% disk./.usageGB‚2719.8134155273438! disk.nvme0n1.in‚ 6970.33203125 disk.dm-0.out‚175076.6171875 network.sent‚ 2.189349e+07 disk.sda1.in‚0 disk.md0.out‚ 1513.9296875! disk.sdb.out‚1553.56298828125" disk.sdb2.out‚1553.56298828125 disk.sda.in‚ 128.6328125( proc.memory.availableMB‚ 1312.9921875! proc.memory.rssMB‚ 2977.140625 disk.rbd2.in‚0" disk.sda2.out‚1553.56298828125 proc.cpu.threads‚17) disk./.usagePercent‚76.04712142849341 disk.md0.in‚ 199.59375 disk.md0p1.out‚ 1513.9296875 disk.rbd0.out‚ 3.087890625 network.recv‚ 3.276306e+06 disk.sda1.out‚0# disk.nvme0n1.out‚123670.0390625 disk.sda2.in‚ 128.6328125 disk.nvme1n1.in‚ 100.4296875* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375! disk.nvme1n1.out‚ 51406.578125 cpu‚25.2081577245011î“Rñ2“$- _wandbruntime‚9935  _runtime‚9935ΣÍÉ”$à ' lr-AdamW/pg1‚0.00019815368342345776 ' lr-AdamW/pg2‚0.00019815368342345776 trainer/global_step‚8299 % _timestamp‚1.7885345521056154e+09 _runtime‚9935.641343892 _step‚678£MèΕ$È _step‚678 ' lr-AdamW/pg1‚0.00019815368342345776 ' lr-AdamW/pg2‚0.00019815368342345776 trainer/global_step‚8299 % _timestamp‚1.7885345521056154e+09 _runtime‚9935.641343892¦r ù.2–$- _wandbruntime‚9935  _runtime‚99357Ã_—$øO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209986535 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697972873987584 RDDevicºn!Úø&eStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153996654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496321202688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707469213193216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209986535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9459903595142 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707468285218816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780081593877504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚770163472689664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496321202688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173188722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674928954129936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9918121187840 % _timestamp‚1.7885345521068664e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173188720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9935.642763579 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103760984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9460811570146 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495411231232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚684389765700082 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚684388840685718 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697972873987584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103760986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707468285218816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210012879 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707469213193216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697972891990528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173188720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674928937090576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697972891990528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153996007 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495411231232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8299 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50235668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚770163496629760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 383175255 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383201601 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856`Pî(˜$ýO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 383175255 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383201601 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209986535 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697972873987584 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153996654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496321202688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707469213193216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209986535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9459903595142 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707468285218816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780081593877504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚770163472689664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496321202688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173188722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674928954129936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9918121187840 % _timestamp‚1.7885345521068664e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173188720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9935.642763579 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103760984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9460811570146 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495411231232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚684389765700082 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚684388840685718 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697972873987584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103760986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707468285218816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210012879 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707469213193216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697972891990528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173188720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674928937090576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697972891990528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153996007 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495411231232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8299 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50235668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚770163496629760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784§çºÔ£2™$- _wandbruntime‚9936  _runtime‚9936Û~œ'š$‚N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚698054511076352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383246618 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675007971724480 _runtime‚9936.863013976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210011366 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103773051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚770253722334208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885345533249571e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173208882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707551953752576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383220246 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698054511076352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103773049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154013745 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9496532691456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675007954685120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707551025764864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210037736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚684469900568112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383246618 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780172979342848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9919280948736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9496532691456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780173236246016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9497442676224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚770253698394112³%ʨ ›$‡N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173208882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50240696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154014395 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154013745 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9496532691456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675007954685120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707551025764864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210037736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚684469900568112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383246618 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚×<²be780172979342848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9919280948736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9496532691456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780173236246016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9497442676224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚770253698394112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚698054511076352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383246618 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675007971724480 _runtime‚9936.863013976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210011366 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103773051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚770253722334208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885345533249571e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173208882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707551953752576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383220246 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698054511076352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103773049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784651377582592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚683 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9515780850250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚711605326935040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174196722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚702053828278784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174196720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚774674781852160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688396522682390 trainer/global_step‚8349 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688395597668026 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154890277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552408627712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154889631 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50525397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976356462592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174196722 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚711605326935040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678879816817776 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385428157 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211257781 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚711606254909440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚774674757912064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678879833857136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551498656256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976595730432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551498656256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104364236 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211257781 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211231437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50526041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚9997.167148734 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784651114374656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9515780850250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚711605326935040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174196722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚702053828278784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174196720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚774674781852160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688396522682390 trainer/global_step‚8349 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688395597668026 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154890277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552408627712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154889631 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50525397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976356462592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174196722 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚711605326935040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678879816817776 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385428157 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211257781 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚711606254909440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚774674757912064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678879833857136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551498656256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976595730432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WEDeviceStatsMonitor.on_train_batch_starÚ7{ít/active_bytes.small_pool.freed‚ 9551498656256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104364236 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211257781 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211231437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50526041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚9997.167148734 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784651114374656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784651377582592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004«OèD2­$- _wandbruntime‚9998  _runtime‚9998×m^œ'®$‚N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚684 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885346148509295e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174216880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784742762858496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977534921728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553530101248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104376301 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154908184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678958851451680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517806098740 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9516898123528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211282638 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688475732535848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚702135447364608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385499520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚702135447364608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50531883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚711688995468800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚711688067481088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174216880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385499520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211282638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚774765007556608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174216882 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211256268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174216882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚702135465367552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154907550 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104376299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688476657550420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211256268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552620116480 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553530101248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚8349Îñl '¯$‡N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174216882 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211256268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174216882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚702135465367552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154907550 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104376299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688476657550420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211256268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552620116480 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553530101248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚8349 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚684 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885346148509295e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174216880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784742762858496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977534921728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553530101248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104376301 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154908184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678958851451680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517806098740 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9516898123528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211282638 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688475732535848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚702135447364608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385499520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚702135447364608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50531883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚711688995468800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚711688067481088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174216880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385499520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211282638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚774765007556608¬°Àü¢2°$- _wandbruntime‚9998  _runtime‚9998ã#Y±$Ó 1loss/SRqqt2/assignment_loss‚1.339877234564887 /loss/FBt1/assignment_loss‚0.396324788141523 1loss/FBt3/assignment_loss‚0.05541032420933334 -loss/FBt2/detection_loss‚2.75508401542902 /loss/FBt3/detection_loss‚0.7119882130684952 /loss/FBt4/detection_loss‚2.0087159434954325 /loss/FRt4/assignment_loss‚2.821177614927292 2loss/SRqqt1/assignment_loss‚1.7677484820286433 /loss/FRt1/detection_loss‚3.1513862816187053 -loss/FRt3/detection_loss‚1.56773486799177 /loss/FRt4/detection_loss‚2.1858598731458185 1loss/SRqqt3/detection_loss‚0.7436711544959018 0loss/FRt3/assignment_loss‚1.0871326800654917 2loss/SRqqt3/assignment_loss‚0.9819494807474817 2loss/SRqqt4/assignment_loss‚1.5727967162217413 1loss/FBt4/assignment_loss‚0.08917797946551786 _runtime‚9998.389287125 trainer/global_step‚8349 1loss/SRqqt2/detection_loss‚2.1862215391463704 /loss/FBt1/detection_loss‚1.9385671530451094 _step‚685 /loss/FRt2/detection_loss‚2.4682193167643547 epoch‚6 /loss/FRt2/assignment_loss‚3.038331177085638 1loss/SRqqt1/detection_loss‚2.3702800571918488 1loss/SRqqt4/detection_loss‚2.0995401305811745 %loss/total_loss‚40.60566868125683 % _timestamp‚1.7885346148539097e+09 /loss/FRt1/assignment_loss‚3.110619673362145 1loss/FBt2/assignment_loss‚0.15785398446314325ð¿Ô^²$Ø 1loss/SRqqt2/assignment_loss‚1.339877234564887 /loss/FBt1/assignment_loss‚0.396324788141523 1loss/FBt3/assignment_loss‚0.05541032420933334 -loss/FBt2/detection_loss‚2.75508401542902 /loss/FBt3/detection_loss‚0.7119882130684952 /loss/FBt4/detection_loss‚2.0087159434954325 /loss/FRt4/assignment_loss‚2.821177614927292 2loss/SRqqt1/assignment_loss‚1.7677484820286433 /loss/FRt1/detection_loss‚3.1513862816187053 -loss/FRt3/detection_loss‚1.56773486799177 /loss/FRt4/detection_loss‚2.1858598731458185 1loss/SRqqt3/detection_loss‚0.7436711544959018 0loss/FRt3/assignment_loss‚1.0871326800654917 2loss/SRqqt3/assignment_loss‚0.9819494807474817 2loss/SRqqt4/assignment_loss‚1.5727967162217413 1loss/FBt4/assignment_loss‚0.08917797946551786 _runtime‚9998.389287125 trainer/global_step‚8349 1loss/SRqqt2/detection_loss‚2.1862215391463704 /loss/FBt1/detection_loss‚1.9385671530451094 _step‚685 /loss/FRt2/detection_loss‚2.4682193167643547 epoch‚6 /loss/FRt2/assignment_loss‚3.038331177085638 1loss/SRqqt1/detection_loss‚2.3702800571918488 1loss/SRqqt4/detection_loss‚2.0995401305811745 %loss/total_loss‚40.60566868125683 % _timestamp‚1.7885346148539097e+09 /loss/FRt1/assignment_loss‚3.110619673362145 1loss/FBt2/assignment_loss‚0.15785398446314325­Ò¦f”³$‚(:‰ ݾëÔýþÜà gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üðÐq´$‚(:æ ݾëÔöí¦â disk.sda.in‚ 128.6328125 disk.sda1.out‚0 disk.nvme1n1.out‚51534 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1523.234375 cpu‚25.206839589996534 disk.sdb.out‚1563.1865234375 disk.sdb1.in‚0 disk.sda.out‚1563.1865234375! disk.sda2.out‚1563.1865234375 disk.md0.out‚ 1523.234375! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 network.recv‚ 3.298564e+06! disk.sdb2.out‚1563.1865234375 disk.sda1.in‚0 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.in‚ 6971.33203125 disk.dm-0.in‚ 7071.76171875! disk.dm-0.out‚175874.26171875 proc.cpu.threads‚17( disk./.usagePercent‚76.0681209673786% disk./.usageGB‚2720.5644607543945 disk.nvme1n1.in‚ 100.4296875$ disk.nvme0n1.out‚124357.76171875 network.sent‚ 2.2036017e+07 disk.sdb2.in‚ 70.9609375 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0$ memory_percent‚91.98713302612305' proc.memory.availableMB‚ 1312.828125P"’×’µ$‚(:‡ ì¾ëÔŸÅà gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Í-"¶$‚(:ö ì¾ëÔŸŠàà# disk.nvme0n1.out‚124534.3984375! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.30424e+06! disk.sda.out‚1575.20263671875 disk.nvme1n1.in‚ 100.4296875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 3.025390625 network.sent‚ 2.213134e+07 disk.md0.in‚ 199.59375 cpu‚25.206538877399858 proc.cpu.threads‚17$ disk./.usageGB‚2720.564582824707! disk.sdb.out‚1575.20263671875# disk.nvme1n1.out‚51534.14453125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 1535.19921875 disk.sda.in‚ 128.6328125! disk.nvme0n1.in‚ 6971.33203125" disk.sda2.out‚1575.20263671875) proc.memory.availableMB‚ 1313.03515625 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 7071.76171875 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0! disk.dm-0.out‚176068.54296875 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1535.19921875$ memory_percent‚91.98586940765381* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06812438051563 disk.sdb2.in‚ 70.9609375" disk.sdb2.out‚1575.20263671875K!“·$‚(:ˆ û¾ëÔªùàà gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùºl~¸$‚(:ó û¾ëÔà«üà proc.cpu.threads‚17" disk.sdb2.out‚1580.08935546875! disk.sda.out‚1580.08935546875 network.sent‚ 2.2139001e+07 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 6986.48828125 disk.sda2.in‚ 128.6328125) disk./.usagePercent‚76.07790952439301% disk./.usageGB‚2720.9145469665527 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 7086.91796875$ memory_percent‚91.98594093322754! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.308e+06 disk.nvme1n1.in‚ 100.4296875* proc.memory.percent‚18.171024322509766 disk.rbd2.in‚0 disk.md0.out‚ 1540.01171875 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1540.01171875! disk.sdb.out‚1580.08935546875# disk.nvme1n1.out‚51633.58984375 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 176555.96875 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd2.out‚ 3.025390625( proc.memory.availableMB‚ 1313.0234375$ disk.nvme0n1.out‚124922.37890625" disk.sda2.out‚1580.08935546875 disk.sda.in‚ 128.6328125 cpu‚25.206279412735306þÌ X¹$‚(:‡ Š¿ëÔ»™»à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.corr´/·†:ectedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÚ¿>ƒº$‚(:ø Š¿ëÔœ•œâ" disk.sda2.out‚1582.24853515625 disk.md0.in‚ 199.59375! disk.sda.out‚1582.24853515625 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.08809699179756 disk.sda.in‚ 128.6328125 network.recv‚ 3.312102e+06! disk.nvme0n1.in‚ 7011.41796875$ disk.nvme0n1.out‚125223.88671875 disk.sdb2.in‚ 70.9609375# disk.nvme1n1.out‚51698.69140625 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1542.12109375 disk.rbd2.out‚ 3.025390625 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 network.sent‚ 2.2144786e+07 disk.dm-0.in‚ 7111.84765625! disk.sdb.out‚1582.24853515625 disk.nvme1n1.in‚ 100.4296875 disk.rbd0.in‚0 cpu‚25.206015504070102 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125 disk.dm-0.out‚ 176922.578125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.out‚ 1542.12109375 disk.sdb.in‚ 70.9609375$ memory_percent‚91.98596477508545% disk./.usageGB‚2721.2789001464844" disk.sdb2.out‚1582.24853515625) proc.memory.availableMB‚ 1313.01953125! proc.memory.rssMB‚ 2977.140625ÎWVN4»$/ _wandbruntime‚10058  _runtime‚10058„aÑʼ$Ä % _timestamp‚1.7885346751509287e+09 _runtime‚10058.686684137 _step‚686 ' lr-AdamW/pg1‚0.00018883613060458908 ' lr-AdamW/pg2‚0.00018883613060458908 trainer/global_step‚8399*/لϽ$É ' lr-AdamW/pg1‚0.00018883613060458908 ' lr-AdamW/pg2‚0.00018883613060458908 trainer/global_step‚8399 % _timestamp‚1.7885346751509287e+09 _runtime‚10058.686684137 _step‚686®“fq4¾$/ _wandbruntime‚10058  _runtime‚10058Õzƒ(¿$úO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚779186067074560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715743296625664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387681059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175204720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9607586081280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212476339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚789220970355200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚715743296625664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚715742368651264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50817278 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212502683 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212476339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682830713584336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155784762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚789221220980224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _runtime‚10058.688596568 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9608496052736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175204722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715742368651264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50817902 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175204720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10035153905664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624õŽ(À$ÿO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚715743296625664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚715742368651264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50817278 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212502683 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212476339 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682830713584336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155784762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚789221220980224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _runtime‚10058.688596568 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9608496052736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175204722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715742368651264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50817902 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175204720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10035153905664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚779186067074560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715743296625664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387681059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175204720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9607586081280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212476339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚789220970355200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¯ƒ g4Á$/ _wandbruntime‚10059  _runtime‚10059W¡7ž 'Â$…N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155802593 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚10059.936258446 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387726050 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682909714139520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706216401655808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715825109197312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175224882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715826037185024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387752422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212501170 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175224880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚779276292779008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885346763978128e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387726050 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212501170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚789312360107008 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175224882 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692482489518156 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10036320036864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706216401655808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9572775378636 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175224880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682909731178880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692483414532728—È‹(±Ã$ŠN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9609617526272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50823044 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pa9Ü_%ool.current‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387752422 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715826037185024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175224882 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692482489518156 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10036320036864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706216401655808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9572775378636 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175224880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682909731178880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692483414532728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155802593 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚10059.936258446 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387726050 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682909714139520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706216401655808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715825109197312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175224882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715826037185024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387752422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212501170 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175224880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚779276292779008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885346763978128e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387726050 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212501170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚789312360107008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213721241 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710215736861184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚793790679275008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176212720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚719879410367488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389960307 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783697352296960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176212720 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156677943 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686781593311536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213721241 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10093571311616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9664583477760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9664583477760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710215754864128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚793790923608576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389933961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10093350918144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176212722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176212722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10119.843455927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105570734 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚696409111632642 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9628443335470 _step‚691 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686781576272176 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176212720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚719879410367488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389960307 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783697352296960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176212720 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156677943 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686781593311536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213721241 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10093571311616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9664583477760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9664583477760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710215754864128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚793790923608576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389933961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10093350918144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176212722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_õbÌôbatch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176212722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10119.843455927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105570734 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚696409111632642 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9628443335470 _step‚691 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686781576272176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213721241 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710215736861184³ˆéäw4Õ$/ _wandbruntime‚10121  _runtime‚10121DÒ–û 'Ö$„N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 390005324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚710297373949952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390005324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629560608956 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9664794966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783787578001408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚10121.066195471 trainer/global_step‚8449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710297373949952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9665704951296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156695770 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9665704951296 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105582799 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10094525670400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚719962150913536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686860610906080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176232882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389978952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710297355947008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176232882 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686860593866720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚719963078901248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51113616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783787554061312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710297355947008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚696490171515036 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176232880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213746072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚696489246500464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚692 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9628652633744ñn’”'×$‰N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51113616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783787554061312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710297355947008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚696490171515036 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176232880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213746072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚696489246500464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚692 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9628652633744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 390005324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚710297373949952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390005324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629560608956 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9664794966528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783787578001408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚10121.066195471 trainer/global_step‚8449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710297373949952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9665704951296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156695770 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9665704951296 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105582799 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10094525670400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚719962150913536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686860610906080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176232882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389978952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710297355947008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176232882 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686860593866720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚719963078901248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2´[Ä4Ø$/ _wandbruntime‚10121  _runtime‚10121u£Ð WÙ$Ñ .loss/FBt2/detection_loss‚2.109804782602522 /loss/FBt3/detection_loss‚0.7270521901251579 % _timestamp‚1.7885347375325341e+09 1loss/SRqqt2/assignment_loss‚1.568825015469509 2loss/SRqqt3/assignment_loss‚1.0306785290689218 .loss/FRt1/detection_loss‚2.970054163652308 0loss/FRt2/assignment_loss‚3.3069720516602197 2loss/SRqqt1/assignment_loss‚1.3079148134396923 0loss/FBt1/assignment_loss‚0.1282275760009952 0loss/SRqqt2/detection_loss‚2.235794014790479 1loss/SRqqt3/detection_loss‚0.9256203417487517 .loss/FBt4/detection_loss‚3.511588290333748 1loss/FBt3/assignment_loss‚0.08598356287587385 %loss/total_loss‚40.91190300165878 epoch‚6 _runtime‚10121.067860549 _step‚693 trainer/global_step‚8449 /loss/FRt3/assignment_loss‚0.988907338216387 /loss/FRt4/assignment_loss‚2.886657659737569 0loss/FBt4/assignment_loss‚0.1297427832826088 1loss/SRqqt4/detection_loss‚2.0077278341936027 /loss/FBt1/detection_loss‚2.2133644342422487 /loss/FRt1/assignment_loss‚3.124924370471169 .loss/FRt2/detection_loss‚2.124637801756358 .loss/FRt3/detection_loss‚1.578604992093711 2loss/SRqqt4/assignment_loss‚1.5353636717342811 1loss/FBt2/assignment_loss‚0.03537930295310884 .loss/FRt4/detection_loss‚2.013755581218679 1loss/SRqqt1/detection_loss‚2.3643218999908817´Þ[¨\Ú$Ö /loss/FBt3/detection_loss‚0.7270521901251579 % _timestamp‚1.7885347375325341e+09 1loss/SRqqt2/assignment_loss‚1.568825015469509 2loss/SRqqt3/assignment_loss‚1.0306785290689218 .loss/FRt1/detection_loss‚2.970054163652308 0loss/FRt2/assignment_loss‚3.3069720516602197 2loss/SRqqt1/assignment_loss‚1.3079148134396923 0loss/FBt1/assignment_loss‚0.1282275760009952 0loss/SRqqt2/detection_loss‚2.235794014790479 1loss/SRqqt3/detection_loss‚0.9256203417487517 .loss/FBt4/detection_loss‚3.511588290333748 1loss/FBt3/assignment_loss‚0.08598356287587385 %loss/total_loss‚40.91190300165878 epoch‚6 _runtime‚10121.067860549 _step‚693 trainer/global_step‚8449 /loss/FRt3/assignment_loss‚0.988907338216387 /loss/FRt4/assignment_loss‚2.886657659737569 0loss/FBt4/assignment_loss‚0.1297427832826088 1loss/SRqqt4/detection_loss‚2.0077278341936027 /loss/FBt1/detection_loss‚2.2133644342422487 /loss/FRt1/assignment_loss‚3.124924370471169 .loss/FRt2/detection_loss‚2.124637801756358 .loss/FRt3/detection_loss‚1.578604992093711 2loss/SRqqt4/assignment_loss‚1.5353636717342811 1loss/FBt2/assignment_loss‚0.03537930295310884 .loss/FRt4/detection_loss‚2.013755581218679 1loss/SRqqt1/detection_loss‚2.3643218999908817 .loss/FBt2/detection_loss‚2.109804782602522µ‹°ƒ“Û$‚(:ˆ Õ¿ëÔijŸà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†¾ |Ü$‚(:ñ Õ¿ëÔ¦¤Šá% disk./.usageGB‚2721.3971252441406 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.nvme1n1.in‚ 100.4296875# disk.nvme1n1.out‚52171.12109375 network.recv‚ 3.334485e+06) disk./.usagePercent‚76.09140261500906 disk.md0.out‚ 1550.23828125 disk.sda.out‚1590.6162109375) proc.memory.availableMB‚ 1312.41015625 cpu‚25.204006158926283! disk.sda2.out‚1590.6162109375 disk.sdb.out‚1590.6162109375! disk.sdb2.out‚1590.6162109375$ memory_percent‚91.98968410491943* proc.memory.percent‚18.171024322509766 disk.rbd2.out‚ 3.025390625 disk.dm-0.in‚ 7163.8984375 disk.md0p1.out‚ 1550.23828125$ disk.nvme0n1.out‚126259.82421875 disk.dm-0.out‚178430.9453125! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.2287098e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sdb1.out‚0 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7063.46875 proc.cpu.threads‚17€/¨“’Ý$‚(:‡ ä¿ëÔƒÙ¦à gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.364' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.591 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûf²ÔêÞ$‚(:ð ä¿ëÔ«êÚà! disk.dm-0.out‚178657.41796875* proc.memory.percent‚18.171024322509766 cpu‚25.20552717965457# disk./.usageGB‚2720.07222366333# disk.nvme1n1.out‚52310.34765625 disk.sdb2.in‚ 70.9609375$ memory_percent‚91.98815822601318 proc.cpu.threads‚17 network.recv‚ 3.339677e+06 disk.md0p1.out‚ 1551.4140625 disk.sda1.out‚0# disk.nvme0n1.out‚126347.0703125 disk.sda.in‚ 128.6328125) proc.memory.availableMB‚ 1312.66015625) disk./.usagePercent‚76.05435781229521 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0" disk.sda2.out‚1591.85791015625 disk.sdb.in‚ 70.9609375! disk.sda.out‚1591.85791015625! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.2382427e+07 disk.dm-0.in‚ 7163.8984375" disk.sdb2.out‚1591.85791015625! disk.sdb.out‚1591.85791015625 disk.md0.in‚ 199.59375 disk.md0.out‚ 1551.4140625 disk.nvme1n1.in‚ 100.4296875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0-y·†‘ disk.sda2.in‚ 128.6328125 disk.sdb1.out‚0 disk.rbd2.out‚ 3.025390625 disk.nvme0n1.in‚ 7063.46875 disk.rbd0.out‚ 3.087890625Ógm’ß$‚(:‡ ó¿ëÔ׋®à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥fUËyà$‚(:î ó¿ëÔ™ûá disk.md0p1.in‚ 199.59375" disk.sda2.out‚1593.68505859375 network.recv‚ 3.343541e+06 disk.sda1.out‚0$ disk.nvme0n1.out‚126347.21484375 disk.rbd2.out‚ 3.025390625 disk.nvme0n1.in‚ 7170.265625 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1553.1796875 disk.sda2.in‚ 128.6328125 disk.sdb1.out‚0 proc.cpu.threads‚17! disk.dm-0.out‚178696.21484375" disk.sdb2.out‚1593.68505859375 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.md0.in‚ 199.59375 disk.rbd0.in‚0* proc.memory.percent‚18.171024322509766 cpu‚25.212001407717842 disk.dm-0.in‚ 7299.74609375 network.sent‚ 2.2387976e+07) disk./.usagePercent‚76.05500086864338% disk./.usageGB‚2720.0952224731445 disk.sdb1.in‚0) proc.memory.availableMB‚ 1312.62890625 disk.md0.out‚ 1553.1796875 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 disk.rbd2.in‚0$ memory_percent‚91.98834896087646! disk.sdb.out‚1593.68505859375 disk.nvme1n1.in‚ 129.48046875 disk.sdb.in‚ 70.9609375! disk.sda.out‚1593.68505859375 disk.nvme1n1.out‚52349¯Ãž±“á$‚(:ˆ ‚ÀëÔ¯†Ãà gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’œŒ tâ$‚(:é ‚ÀëÔÉ™ßá! disk.dm-0.out‚178760.01171875 disk.sda1.in‚0 disk.dm-0.in‚ 7307.0625 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125! disk.sdb2.out‚1595.1279296875! disk.sda2.out‚1595.1279296875 disk.sdb.out‚1595.1279296875 disk.sda.in‚ 128.6328125 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 2.2394892e+07) disk./.usagePercent‚76.05594268114201 disk./.usageGB‚ 2720.12890625 network.recv‚ 3.346544e+06 disk.sdb1.in‚0 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.out‚ 126410.84375 disk.sda.out‚1595.1279296875 disk.md0p1.in‚ 199.59375 disk.nvme1n1.in‚ 129.50390625# disk.nvme1n1.out‚52349.16796875 disk.sdb1.out‚0) proc.memory.availableMB‚ 1312.62109375 disk.md0.out‚ 1554.56640625 disk.sdb2.in‚ 70.9609375$ memory_percent‚91.98839664459229! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.2177677345607! disk.nvme0n1.in‚ 7177.55859375 disk.md0.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.md0p1.out‚ 1554.56640625x¼Ü¸’ã$‚(:‡ ‘ÀëÔ×®à gpu.0.gpu‚13.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.59' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä'örä$‚(:ç ‘ÀëÔ¾µâà disk.md0.in‚ 199.59375 network.recv‚ 3.351052e+06 disk.rbd2.out‚ 3.025390625" disk.nvme0n1.out‚ 126549.296875$ memory_percent‚91.98837280273438! proc.memory.rssMB‚ 2977.140625 disk.md0.out‚ 1556.08984375 disk.dm-0.out‚ 178898.953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0! disk.sda2.out‚1596.7119140625 disk.sda.out‚1596.7119140625 disk.rbd0.out‚ 3.087890625 disk.sda1.out‚0! disk.nvme0n1.in‚ 7177.58984375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1556.08984375$ proc.memory.availableMB‚1312.625) disk./.usagePercent‚76.05952487511331$ disk./.usageGB‚2720.257022857666! disk.sdb2.out‚1596.7119140625 disk.nvme1n1.out‚ 52349.65625 proc.cpu.threads‚17 disk.sda2.in‚ 128.6328125 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.223966663079995 disk.nvme1n1.in‚ 129.50390625 network.sent‚ 2.2398896e+07 disk.dm-0.in‚ 7307.09375 disk.sdb.out‚1596.7119140625 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375* proc.memory.percent‚18.171024322509766¹7*4å$/ _wandbruntime‚10191  _runtime‚10191ÖIóe*æ$ßT K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391759213 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10134016873984 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 177253284 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 214505929 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚797851615228928 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9703619335680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚213 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 162113024 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 391779926 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 177253286 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 177253284 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚714017188506112 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚215 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 157653987 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚51445805 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚797851801282048 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚700026045274698 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9704302374912 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚787717622295040 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 214526640 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚714017206509056 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚714017206509056 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 214526640 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9703619335680 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚690358701749808 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚51446018 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚690358718789168 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚700026743381506 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚723720807841792 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚787717598354944 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9667343524890 " _timestamp‚1.7885348079413e+09jíëÞj*ç$äT OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157654202 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 177253286 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 106208182 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 186053120 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚10191.477293674 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 214505929 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚714017188506112 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚215 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 157653987 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚51445805 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚797851801282048 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚700026045274698 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9704302374912 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚787717622295040 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 214526640 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚714017206509056 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚714017206509056 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 214526640 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9703619335680 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚690358701749808 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚51446018 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚690358718789168 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚700026743381506 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚723720807841792 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚787717598354944 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9667343524890 " _timestamp‚1.7885348079413e+09 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391759213 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10134016873984 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 177253284 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 214505929 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚797851615228928 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9703619335680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚213 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 162113024 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 391779926 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 177253286 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 177253284 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 186053120 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚714071449413120 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚51451297 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 _runtime‚10192.530807205 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 177268760 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚723776259641856 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9704792225792 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚8462 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚787773889848832 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 106217746 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 177268762 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 177268760 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚723775558599680 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 214535454 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 _Ñ\¤RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 162113024 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 214535454 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9704109186560 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚51451082 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885348089949095e+09 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 186053120 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚714071449413120 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚51451297 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 _runtime‚10192.530807205 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 177268760 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚723776259641856 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9704792225792 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚8462 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚787773889848832 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 106217746 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 177268762 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 177268760 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚723775558599680 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 214535454 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 162113024 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 214535454 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9704109186560 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚51451082 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885348089949095e+09 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 Bî$ñ¦ ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25642532110214233 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947901725769 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44927337765693665 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2556770443916321 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_2_of_5‚0.4747631847858429 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16847839951515198 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.022715052589774132 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05386904627084732 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3441394567489624 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.543829619884491 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25232458114624023 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5070860385894775 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 (particle/accuracy‚0.8520534634590149 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7067073583602905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02449933998286724 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5310077667236328 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3575892746448517 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.407625287771225 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5412890315055847 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2534482777118683 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010321609675884247 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9878968596458435 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5246913433074951 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28733596205711365 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1884920597076416 particle/accuracy_3_of_6‚1 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001873897621408105 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49117401242256165 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3175926208496094 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_4_of_7‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03104575164616108 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.011290322057902813 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 'particle/f_score‚0.4066488444805145 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4814237058162689 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_3_of_4‚0.08398382365703583 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1626984030008316 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29357802867889404 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4871794581413269 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4763597249984741 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02210884355008602 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912018775939941 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21780000627040863 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,jet/accuracy_4_of_5‚0.013978494331240654 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 jet/accuracy_1_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45568138360977173 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47769519686698914 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24228395521640778 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2075755000114441 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5086207389831543 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4597076177597046 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424242496490479 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19524447619915009 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5437788367271423 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1763318032026291 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 jet/accuracy_7_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1799156218767166 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15347065031528473 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.49792030453681946 particle/accuracy_2_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_1_of_1‚0.45723143219947815 particle/accuracy_3_of_4‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4704509377479553 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5149891376495361 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647272527217865 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.011290322057902813 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005866377614438534 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02392873540520668 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9863446950912476 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6307073831558228 +jet/accuracy_3_of_5‚0.12122374773025513 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872754216194153 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.274353563785553 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893332719802856 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4836626350879669 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9920201301574707 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5466552376747131 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5457595586776733 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38955849409103394 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19065655767917633 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_5_of_7‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2041393220424652 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45748475193977356 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44111278653144836 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6009432673454285 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.646726131439209 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 )loss/val_total_loss‚40.41571997690851 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20561246573925018 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2604166865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899207353591919 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991100549697876 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15761633217334747 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985594630241394 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748656153678894 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4586309790611267 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 6 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.182758629322052 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4333076477050781 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2668578624725342 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10468290001153946 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1808510571718216 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19317464530467987 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5905998945236206 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047529201954603195 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5910087823867798 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3C˜ŠºþT_of_3‚0.031084034591913223 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5732715725898743 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10836850851774216 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5657640099525452 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44977012276649475 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5016984343528748 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929622411727905 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_4‚0.9236872792243958 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3188542127609253 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788534823212992e+09 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.565839946269989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5136759281158447 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 _runtime‚10206.750807502 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923873543739319 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4333076477050781 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3349567949771881 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚8462 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.24192039668560028 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19323977828025818 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055989041924476624 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24247801303863525 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984128475189209 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927942156791687 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5439221262931824 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45781534910202026 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2552015483379364 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06262117624282837 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02600781060755253 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21585755050182343 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166269302368 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45754513144493103 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1805732101202011 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4893518388271332 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5039548277854919 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2142857164144516 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978939294815063 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19291777908802032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4704116880893707 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46283528208732605 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4436781704425812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9927274584770203 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.20614035427570343 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5692840814590454 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4755682945251465 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8176921606063843 particle/accuracy_1_of_2‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5040404796600342 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03104575164616108 particle/accuracy_4_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16497910022735596 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014789619483053684 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3817708492279053 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15895062685012817 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17844784259796143 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05380868911743164 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19174693524837494 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.11538461595773697 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5371294021606445 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11428507417440414 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4163194000720978 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.66865074634552 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44754645228385925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18219536542892456 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5444139242172241 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.48033037781715393 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860754013061523 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5312499403953552 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42402002215385437 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44766315817832947 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41699346899986267 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913750290870667 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7500616312026978 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1862093061208725 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2425553947687149 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3808581233024597 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16899381577968597 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25270989537239075 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.958026111125946 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1862093061208725 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19111110270023346 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3483252227306366 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_4‚0.42283833026885986 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24299356341362 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17816093564033508 +jet/accuracy_2_of_2‚0.11493827402591705 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚6 _step‚696 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3221355974674225 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46029341220855713 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025059157982468605 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.006133378949016333 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1812865436077118 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.44756340980529785 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.347375750541687 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 *jet/accuracy_1_of_3‚0.8591517210006714 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4583141803741455 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46244946122169495 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01637629047036171 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39074069261550903 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0398193821310997 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08537034690380096 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17004504799842834 +particle/specificity‚0.9496130347251892 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43299856781959534 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5545138120651245 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19677363336086273 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.32490745186805725 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23717768490314484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4703391194343567 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026158401742577553 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22927920520305634 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5553098917007446 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_1_of_1‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36077722907066345 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6validation_average_jet_accuracy‚0.4262423813343048 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3983535170555115 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2941841185092926 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19227513670921326 particle/accuracy_1_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4504302442073822 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18261456489562988 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_1_of_6‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.19667658209800726Ýnô*ï$ö¦ ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23717768490314484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4703391194343567 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026158401742577553 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22927920520305634 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5553098917007446 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_1_of_1‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36077722907066345 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6validation_average_jet_accuracy‚0.4262423813343048 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3983535170555115 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2941841185092926 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19227513670921326 particle/accuracy_1_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4504302442073822 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18261456489562988 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_1_of_6‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25642532110214233 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947901725769 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44927337765693665 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2556770443916321 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_2_of_5‚0.4747631847858429 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16847839951515198 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.022715052589774132 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05386904627084732 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3441394567489624 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.543829619884491 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25232458114624023 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5070860385894775 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 (particle/accuracy‚0.8520534634590149 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7067073583602905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02449933998286724 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5310077667236328 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3575892746448517 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.407625287771225 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5412890315055847 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2534482777118683 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010321609675884247 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9878968596458435 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5246913433074951 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28733596205711365 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1884920597076416 particle/accuracy_3_of_6‚1 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001873897621408105 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49117401242256165 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3175926208496094 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_4_of_7‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03104575164616108 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.011290322057902813 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 'particle/f_score‚0.4066488444805145 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4814237058162689 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_3_of_4‚0.08398382365703583 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1626984030008316 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29357802867889404 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4871794581413269 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4763597249984741 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02210884355008602 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912018775939941 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21780000627040863 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,jet/accuracy_4_of_5‚0.013978494331240654 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 jet/accuracy_1_of_7‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45568138360977173 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47769519686698914 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24228395521640778 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2075755000114441 ,&Puri§õYG‰hty/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5086207389831543 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4597076177597046 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424242496490479 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19524447619915009 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5437788367271423 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1763318032026291 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 jet/accuracy_7_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1799156218767166 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15347065031528473 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.49792030453681946 particle/accuracy_2_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_1_of_1‚0.45723143219947815 particle/accuracy_3_of_4‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4704509377479553 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5149891376495361 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647272527217865 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.011290322057902813 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005866377614438534 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02392873540520668 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9863446950912476 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6307073831558228 +jet/accuracy_3_of_5‚0.12122374773025513 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872754216194153 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.274353563785553 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893332719802856 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4836626350879669 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9920201301574707 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5466552376747131 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5457595586776733 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38955849409103394 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19065655767917633 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_5_of_7‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2041393220424652 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45748475193977356 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44111278653144836 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6009432673454285 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.646726131439209 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 )loss/val_total_loss‚40.41571997690851 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20561246573925018 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2604166865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899207353591919 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991100549697876 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15761633217334747 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985594630241394 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748656153678894 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4586309790611267 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 6 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.182758629322052 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4333076477050781 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2668578624725342 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10468290001153946 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1808510571718216 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19317464530467987 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5905998945236206 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047529201954603195 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5910087823867798 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.031084034591913223 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5732715725898743 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10836850851774216 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5657640099525452 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44977012276649475 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5016984343528748 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929622411727905 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_4‚0.9236872792243958 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3188542127609253 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788534823212992e+09 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.565839946269989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5136759281158447 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 _runtime‚10206.750807502 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923873543739319 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4333076477050781 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3349567949771881 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚8462 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.24192039668560028 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19323977828025818 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055989041924476624 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24247801303863525 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984128475189209 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927942156791687 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5439221262931824 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45781534910202026 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2552015483379364 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06262117624282837 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02600781060755253 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21585755050182343 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166269302368 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45754513144493103 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1805732101202011 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4893518388271332 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5039548277854919 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2142857164144516 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978939294815063 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19291777908802032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4704116880893707 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46283528208732605 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4436781704425812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9927274584770203 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.20614035427570343 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5692840814590454 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4755682945251465 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8176921606063843 particle/accuracy_1_of_2‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5040404796600342 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03104575164616108 particle/accuracy_4_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16497910022735596 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014789619483053684 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3817708492279053 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15895062685012817 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17844784259796143 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05380868911743164 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19174693524837494 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.11538461595773697 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5371294021606445 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11428507417440414 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4163194000720978 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.66865074634552 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44754645228385925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18219536542892456 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5444139242172241 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.48033037781715393 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9860754013061523 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5312499403953552 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42402002215385437 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44766315817832947 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41699346899986267 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913750290870667 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7500616312026978 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1862093061208725 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2425553947687149 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3808581233024597 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16899381577968597 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25270989537239075 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.958026111125946 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1862093061208725 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19111110270023346 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3483252227306366 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_2_of_4‚0.42283833026885986 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24299356341362 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17816093564033508 +jet/accuracy_2_of_2‚0.11493827402591705 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚6 _step‚696 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3221355974674225 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46029341220855713 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025059157982468605 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.006133378949016333 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1812865436077118 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.44756340980529785 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.347375750541687 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 *jet/accuracy_1_of_3‚0.8591517210006714 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4583141803741455 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46244946122169495 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01637629047036171 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39074069261550903 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0398193821310997 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08537034690380096 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17004504799842834 +particle/specificity‚0.9496130347251892 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43299856781959534 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5545138120651245 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19677363336086273 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.32490745186805725 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192¸Ê‘ i”ð$‚(:‰ ¯ÀëÔ÷™Ÿá gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} óÏnñ$‚(:ã ¯ÀëÔÄÔóá disk.rbd0.out‚ 3.087890625 disk.sda1.out‚0 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚1600.361328125 disk.nvme1n1.out‚ 52351.0625 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0" disk.nvme0n1.out‚ 126823.234375 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1559.609375 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1600.361328125 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sda2.out‚1600.361328125 disk.dm-0.in‚ 7307.09375 network.sent‚ 2.2512242e+07 network.recv‚ 3.360209e+06$ disk./.usageGB‚2720.393253326416 disk.sda.out‚1600.361328125 disk.rbd2.in‚0 disk.dm-0.out‚ 179174.296875 disk.sda1.in‚0! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.233266524650794) proc.memory.availableMB‚ 1312.32421875 disk.md0p1.out‚ 1559.609375 disk.sdb1.out‚0! disk.nvme0n1.in‚ 7177.58984375$ memory_percent‚91.99020862579346) disk./.usagePercent‚76.06333393603644 disk.nvme1n1.in‚ 129.50390625 disk.rbd2.out‚ 3.025390625 disk.md0.in‚ 199.59375 proc.cpu.threads‚17…8g”ò$‚(:‰ ¾ÀëÔЗ›à gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Ïmnó$‚(:ô ¾ÀëÔ´®Ïà network.recv‚ 3.365321e+06" disk.sdb2.out‚1601.81591796875! disk.sda.out‚1601.81591796875 disk.sdb1.out‚0$ memory_percent‚91.99023246765137" disk.sda2.out‚1601.81591796875! disk.dm-0.out‚179379.39453125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 129.50390625 disk.dm-0.in‚ 7307.09375! disk.sdb.out‚1601.81591796875! disk.nvme0n1.in‚ 7177.58984375$ disk.nvme0n1.out‚127028.19921875 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625* proc.memory.percent‚18.171024322509766% disk./.usageGB‚2720.1184883117676 disk.sda1.out‚0 disk.md0p1.out‚ 1561.0078125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 network.sent‚ 2.2637426e+07 disk.md0.in‚ 199.59375 disk.md0.out‚ 1561.0078125" disk.nvme1n1.out‚ 52351.1953125 disk.sdb1.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.05565139122878 disk.md0p1.in‚ 199.59375( proc.memory.availableMB‚ 1312.3203125 cpu‚25.232980674895277 proc.cpu.threads‚17«2ÒV“ô$‚(:ˆ ÍÀëԺܨâ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bA]Övõ$‚(:ë ÍÀëỔ—ã network.recv‚ 3.369942e+06 disk.sdb.out‚1603.0908203125" disk.nvme1n1.out‚ 52351.4765625* proc.memory.percent‚18.171024322509766! disk.sdb2.out‚1603.0908203125 disk.sda.out‚1603.0908203125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚127070.6171875 disk.md0.in‚ 199.59375( proc.memory.availableMB‚ 1312.3203125! proc.memory.rssMB‚ 2977.140625 disk.sda.in‚ 128.6328125 disk.sda1.out‚0! disk.sda2.out‚1603.0908203125 disk.nvme1n1.in‚ 129.50390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 3.025390625 cpu‚25.23270025951401 network.sent‚ 2.2641364e+07 disk.sdb.in‚ 70.9609375 disk.md0p1.out‚ 1562.2265625! disk.nvme0n1.in‚ 7177.58984375 disk.dm-0.out‚ 179422.09375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375$ memory_percent‚91.99023246765137) disk./.usagePercent‚76.05576103825581% disk./.usageGB‚2720.1224098205566 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1562.2265625 disk.dm-0.in‚ 7307.09375¤°ÂJ4ö$/ _wandbruntime‚10248  _runtime‚10248¤W\ÔÊ÷$Ä _step‚697 ' lr-AdamW/pg1‚0.00017966616925698214 ' lr-AdamW/pg2‚0.00017966616925698214 trainer/global_step‚8499 % _timestamp‚1.7885348652789478e+09 _runtime‚10248.814611122ÉC2Ïø$É ' lr-AdamW/pg2‚0.00017966616925698214 trainer/global_step‚8499 % _timestamp‚1.7885348652789478e+09 _runtime‚10248.814611122 _step‚697 ' lr-AdamW/pg1‚0.00017966616925698214¹C㫶4ù$/ _wandbruntime‚10248  _runtime‚10248ú=Œú$ûO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178195708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_Ií3ý$batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚717715146296832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178195710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10183185112064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178195708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚717715128293888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9750638524928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393719065 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791753469468160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9715094983643 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727465766818816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9751548496384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791753493408256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9751548496384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215549701 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106776390 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801936421603840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51723519 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727466694793216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717715146296832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215523357 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚10248.816132957 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9714187008639 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885348652801685e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215523357 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚693909317665328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178195710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106776392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158499304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚717715128293888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152Ï#, (û$€P ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9751548496384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215549701 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106776390 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801936421603840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51723519 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727466694793216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717715146296832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215523357 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚10248.816132957 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9714187008639 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885348652801685e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215523357 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚693909317665328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178195710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106776392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158499304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚717715128293888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178195708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚717715146296832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178195710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10183185112064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178195708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚717715128293888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9750638524928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393719065 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791753469468160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9715094983643 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727465766818816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9751548496384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791753493408256ºÛ–Ç4ü$/ _wandbruntime‚10250  _runtime‚10250<¢Ò0 'ý$…N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393790428 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178215870 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚717796765382656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51728588 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10184116920320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393764056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215574558 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158517645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51729188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9715304281917 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885348664822078e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393790428 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚717796765382656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178215868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791843719112704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215548188 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚717796747379712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9716212257129 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727549435352576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10184347786240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693988335259872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802028066898944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727548507364864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158517043 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693988318220512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 õˆ×› þ$ŠN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106788457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9752669969920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9751759985152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9752669969920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106788455 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158517043 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693988318220512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393790428 OADeviceStatsMonitor.on_trlDóâuain_batch_end/active.large_pool.allocated‚ 178215870 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚717796765382656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51728588 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10184116920320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393764056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215574558 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158517645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51729188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9715304281917 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885348664822078e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393790428 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚717796765382656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178215868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791843719112704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215548188 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚717796747379712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9716212257129 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727549435352576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10184347786240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693988335259872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802028066898944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727548507364864 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796264778630656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52007514 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179203708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796264754690560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707631169631279 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚721796082585088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9807635921408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9770972238751 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9770064263747 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107379640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚806505837002752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚697860197392528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731602808535040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179203710 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9806725949952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚721796100588032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52006891 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107379642 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚8549 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216768259 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707630244616915 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731602808535040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9806725949952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179203710 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179203708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216768259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10241082312192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159386531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697860180353168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395971967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784¤ž|^Ù%€P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796264754690560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707631169631279 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚721796082585088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9807635921408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9770972238751 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9770064263747 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107379640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚806505837002752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚697860197392528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731602808535040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179203710 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9806725949952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚721796100588032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52006891 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107379642 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚8549 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216768259 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707630244616915 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731602808535040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9806725949952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179203710 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179203708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216768259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10241082312192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159386531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697860180353168 MGDeviceStatsMonitor.on_train_batch_start/reservƒßÚ-ed_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395971967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796264778630656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52007514 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179203708¾ÄŒ¸ä4%/ _wandbruntime‚10310  _runtime‚10310g '‘%…N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396043330 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9807847410176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚10310.697340113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159404949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52012607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9808757394944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107391707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚721877701670912 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179223870 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731686477068800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216793090 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9807847410176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806597500064256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707710379484737 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9808757394944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697939197947712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731686477068800 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚721877719673856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179223870 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731685549081088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216819460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396016958 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179223868 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216819460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52013242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697939214987072 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚721877701670912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806597257841152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707711304499309 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179223868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159404312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396043330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396016958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216793090 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10242277446144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚703 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚796354980395008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885349271595533e+09N‚ÎÝ'’%ŠN B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚721877701670912 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179223870 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731686477068800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216793090 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9807847410176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806597500064256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707710379484737 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9808757394944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697939197947712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731686477068800 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚721877719673856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179223870 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731685549081088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216819460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396016958 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179223868 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216819460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52013242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697939214987072 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚721877701670912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806597257841152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707711304499309 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179223868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159404312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396043330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396016958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216793090 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10242277446144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚703 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚796354980395008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885349271595533e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396043330 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9807847410176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚10310.697340113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159404949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52012607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9808757394944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107391707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¿0žŠa4“%/ _wandbruntime‚10310  _runtime‚10310yÝa_”%Ù 1loss/FBt2/assignment_loss‚0.21300608574955296 .loss/FRt1/detection_loss‚2.838431994704639 .loss/FRt3/detection_loss‚1.525932988555481 0loss/SRqqt4/detection_loss‚2.005050364562443 /loss/FRt4/detection_loss‚2.4910791714675726 /loss/FBt3/detection_loss‚0.4834446785012808 /loss/FBt4/detection_loss‚2.2796786725521088 epoch‚7 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚2.7781001680037556 1loss/SRqqt1/detection_loss‚2.3132794331759214 1loss/SRqqt3/detection_loss‚0.9096525789958969 /loss/FBt1/detection_loss‚2.9974018004205494 /loss/FBt2/detection_loss‚1.8856020674985998 _runtime‚10310.698708469 2loss/SRqqt3/assignment_loss‚0.7530034856633547 % _timestamp‚1.7885349271633399e+09 0loss/FRt3/assignment_loss‚1.2093306005001068 0loss/FRt4/assignment_loss‚2.9687498286366463 2loss/SRqqt2/assignment_loss‚1.7308857454253095 2loss/SRqqt4/assignment_loss‚1.1863976557872125 1loss/FBt1/assignment_loss‚0.07945271834098043 0loss/FBt4/assignment_loss‚0.0494516404537535 &loss/total_loss‚39.754766345156995 1loss/FBt3/assignment_loss‚0.19426360571934367 _step‚704 0loss/FRt2/assignment_loss‚2.8004594908820257 1loss/SRqqt1/assignment_loss‚1.404903985094279 /loss/FRt2/detection_loss‚2.4023807215689885 0loss/SRqqt2/detection_loss‚2.254826862897192ø©M6d•%Þ 1loss/FBt2/assignment_loss‚0.21300608574955296 .loss/FRt1/detection_loss‚2.838431994704639 .loss/FRt3/detection_loss‚1.525932988555481 0loss/SRqqt4/detection_loss‚2.005050364562443 /loss/FRt4/detection_loss‚2.4910791714675726 /loss/FBt3/detection_loss‚0.4834446785012808 /loss/FBt4/detection_loss‚2.2796786725521088 epoch‚7 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚2.7781001680037556 1loss/SRqqt1/detection_loss‚2.3132794331759214 1loss/SRqqt3/detection_loss‚0.9096525789958969 /loss/FBt1/detection_loss‚2.9974018004205494 /loss/FBt2/detection_loss‚1.8856020674985998 _runtime‚10310.698708469 2loss/SRqqt3/assignment_loss‚0.7530034856633547 % _timestamp‚1.7885349271633399e+09 0loss/FRt3/assignment_loss‚1.2093306005001068 0loss/FRt4/assignment_loss‚2.9687498286366463 2loss/SRqqt2/assignment_loss‚1.7308857454253095 2loss/SRqqt4/assignment_loss‚1.1863976557872125 1loss/FBt1/assignment_loss‚0.07945271834098043 0loss/FBt4/assignment_loss‚0.0494516404537535 &loss/total_loss‚39.754766345156995 1loss/FBt3/assignment_loss‚0.19426360571934367 _step‚704 0loss/FRt2/assignment_loss‚2.8004594908820257 1loss/SRqqt1/assignment_loss‚1.404903985094279 /loss/FRt2/detection_loss‚2.4023807215689885 0loss/SRqqt2/detection_loss‚2.254826862897192Àx Ó–”–%‚(:‰ ˜ÁëÔÀÞ™à gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QHð{—%‚(:ð ˜ÁëÔÿÊá network.recv‚ 3.392932e+06) disk./.usagePercent‚76.09158628444538 disk.sdb.out‚1611.5947265625 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0! disk.sda2.out‚1611.5947265625$ memory_percent‚91.99268817901611 disk.dm-0.in‚ 7307.09375 disk.sda.out‚1611.5947265625 disk.rbd0.out‚ 3.087890625) proc.memory.availableMB‚ 1311.91796875! proc.memory.rssMB‚ 2977.140625 cpu‚25.23116646894213! disk.sdb2.out‚1611.5947265625 disk.nvme1n1.in‚ 129.50390625 disk.sda1.in‚0 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚52352.66796875 disk.sda2.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 disk.rbd2.out‚ 3.025390625" disk.nvme0n1.out‚ 127859.515625$ disk./.usageGB‚2721.403694152832 disk.md0.in‚ 199.59375 network.sent‚ 2.2785903e+07 disk.sdb1.out‚0! disk.nvme0n1.in‚ 7177.58984375! disk.dm-0.out‚180212.18359375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.out‚ 1570.41796875 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1570.41796875 disk.sda1.out‚0 proc.cpu.threads‚17ÂdŽ"’˜%‚(:‡ §ÁëÔúßÃà gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îWýn™%‚(:ã §ÁëÔºß÷à! disk.nvme0n1.in‚ 7177.58984375 disk.sda2.out‚ 1613.2109375 disk.sdb.out‚ 1613.2109375 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375# disk.nvme1n1.out‚52352.91015625 cpu‚25.230883009572636% disk./.usageGB‚2721.4016189575195# disk.nvme0n1.out‚128118.2109375 disk.sdb2.in‚ 70.9609375 disk.sdb2.out‚ 1613.2109375 disk.nvme1n1.in‚ 129.50390625 disk.rbd2.out‚ 3.025390625 disk.sda1.in‚0! disk.dm-0.out‚180465.24609375! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.397842e+06 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1571.9765625 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1571.9765625 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.sda.out‚ 1613.2109375$ memory_percent‚91.99271202087402 network.sent‚ 2.2882377e+07 disk.md0p1.in‚ 199.59375( proc.memory.availableMB‚ 1311.9140625 proc.cpu.threads‚17( disk./.usagePercent‚76.0915282611159 disk.rbd0.in‚0 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 7307.09375* proc.memory.percent‚18.171024322509766á ±[”š%‚(:‰ ¶ÁëÔÿ£ à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â,u›%‚(:ê ¶ÁëÔŸì¯á) disk./.usagePercent‚76.09152986102389 disk.sdb1.out‚0 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1573.69921875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17% disk./.usageGB‚2721.4016761779785 disk.sdb1.in‚0 disk.md0.in‚ 199.59375 disk.md0p1.in‚ 199.59375! disk.nvme0n1.in‚ 7177.58984375 disk.sda2.in‚ 128.6328125 disk.sdb.out‚ 1615.00390625 network.recv‚ 3.401907e+06 disk.sda2.out‚ 1615.00390625 disk.md0.out‚ 1573.69921875 disk.sda.out‚ 1615.00390625$ memory_percent‚91.99423789978027 network.sent‚ 2.2888186e+07$ disk.nvme0n1.out‚128337.19921875 disk.dm-0.in‚ 7307.09375! disk.dm-0.out‚180690.26953125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 129.50390625 cpu‚25.230595168978184 disk.sda1.in‚0 disk.rbd2.out‚ 3.025390625 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1615.00390625" disk.nvme1n1.out‚ 52353.0703125( proc.memory.availableMB‚ 1311.6640625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625! proc.memory.rssMB‚ 2977.140625¤åFï’œ%‚(:‡ ÅÁëÔÚë¥à gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öœïvz%‚(:è ÅÁëÔƒê‚á disk.md0.out‚ 1575.06640625 disk.rbd0.out‚ 3.087890625$ memory_percent‚91.99271202087402! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.nvme1n1.out‚ 52353.34375 network.recv‚ 3.406855e+06! disk.nvme0n1.in‚ 7177.58984375 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0 disk.sda2.in‚ 128.6ç6Èù328125 disk.rbd0.in‚0( proc.memory.availableMB‚ 1311.9140625 cpu‚25.230288063501355 network.sent‚ 2.2894403e+07% disk./.usageGB‚2721.1245880126953 disk.md0p1.out‚ 1575.06640625 disk.rbd2.out‚ 3.025390625" disk.nvme0n1.out‚ 128548.109375 disk.sda2.out‚1616.439453125 disk.sdb2.out‚1616.439453125 disk.md0p1.in‚ 199.59375) disk./.usagePercent‚76.08378235995215 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.nvme1n1.in‚ 129.50390625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 7307.09375 disk.sdb.out‚1616.439453125 disk.sdb2.in‚ 70.9609375 disk.dm-0.out‚ 180901.453125 disk.sda.out‚1616.439453125 proc.cpu.threads‚17¶ÑDö4ž%/ _wandbruntime‚10370  _runtime‚10370–´õÊŸ%Ä ' lr-AdamW/pg2‚0.00017065144300718813 trainer/global_step‚8599 % _timestamp‚1.7885349868771598e+09 _runtime‚10370.412801165 _step‚705 ' lr-AdamW/pg1‚0.00017065144300718813á`éÏ %É % _timestamp‚1.7885349868771598e+09 _runtime‚10370.412801165 _step‚705 ' lr-AdamW/pg1‚0.00017065144300718813 ' lr-AdamW/pg2‚0.00017065144300718813 trainer/global_step‚8599Á\€144¡%/ _wandbruntime‚10370  _runtime‚10370Ý µ<(¢%ûO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735739850251264 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107982892 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚706 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218039505 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725877054879232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701811077119728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚735740778225664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218013161 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180211710 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725877054879232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9826849493859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107982890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52291938 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52291311 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735740778225664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218039505 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180211708 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160274201 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚735739850251264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701811060080368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9825941518855 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725877036876288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615^,ôt(£%€P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218013161 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180211710 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725877054879232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9826849493859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107982890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52291938 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52291311 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735740778225664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218039505 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180211708 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160274201 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚735739850251264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701811060080368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9825941518855 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725877036876288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735739850251264 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107982892 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚706 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218039505 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725877054879232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701811077119728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚735740778225664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ˈ4¤%/ _wandbruntime‚10371  _runtime‚10371j¥ÔÛ '¥%…N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52296993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚811166993994752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚711717136467045 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180231870 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180231868 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398296232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10300467279872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218064362 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885349880807521e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10300704437248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218037992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚735823518785024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107994955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864844819968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725958655962112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160291948 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160292571 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725958655962112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚735822590797312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811166732897280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9827966767345 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701890094714272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚735823518785024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚711718061481617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9827058792133 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398269860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52297614 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398269860 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚701890077674912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398296232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725958673965056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107994957 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180231870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725958673965056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9863934835200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218064362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚10371.618075163G½Ü†z¦%ŠN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218064362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚10371.618075163 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_byŠü§…–#tes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52296993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚811166993994752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚711717136467045 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180231870 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180231868 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398296232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10300467279872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218064362 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885349880807521e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10300704437248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218037992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚735823518785024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107994955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864844819968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725958655962112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160291948 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160292571 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725958655962112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚735822590797312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811166732897280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9827966767345 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701890094714272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚735823518785024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚711718061481617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9827058792133 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398269860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52297614 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398269860 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚701890077674912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398296232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725958673965056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107994957 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180231870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚725958673965056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9863934835200Ãýù 4§%/ _wandbruntime‚10371  _runtime‚10371L†y~\¨%Ö 2loss/SRqqt4/assignment_loss‚1.2017948126109939 .loss/FRt4/detection_loss‚2.281179890284936 1loss/SRqqt3/detection_loss‚0.8514861279389764 2loss/SRqqt3/assignment_loss‚0.9302830391324464 epoch‚7 trainer/global_step‚8599 0loss/FRt1/assignment_loss‚2.3120882702725276 0loss/FBt1/assignment_loss‚0.3881527739285957 0loss/FBt2/assignment_loss‚0.2104809042425142 /loss/FRt3/detection_loss‚1.3549178776937345 1loss/SRqqt2/detection_loss‚2.4225667532413238 /loss/FBt3/detection_loss‚0.8169539699572888 /loss/FRt4/assignment_loss‚2.781644711229536 1loss/FBt3/assignment_loss‚0.11984926721578418 1loss/FBt4/assignment_loss‚0.10966525430871325 .loss/FRt1/detection_loss‚2.980973007423537 .loss/FRt2/detection_loss‚2.310784518798755 2loss/SRqqt1/assignment_loss‚1.2776837163737842 1loss/SRqqt1/detection_loss‚2.3348636167390007 .loss/FBt2/detection_loss‚2.156221214681864 1loss/SRqqt4/detection_loss‚2.0213512517511845 % _timestamp‚1.7885349880841556e+09 _runtime‚10371.61960442 _step‚708 .loss/FBt1/detection_loss‚2.355678152292967 /loss/FBt4/detection_loss‚1.8894296448964338 &loss/total_loss‚38.559273754804444 0loss/FRt2/assignment_loss‚2.9938911980106715 2loss/SRqqt2/assignment_loss‚1.1669205506482432 0loss/FRt3/assignment_loss‚1.29041323113063026k×na©%Û 0loss/FRt3/assignment_loss‚1.2904132311306302 2loss/SRqqt4/assignment_loss‚1.2017948126109939 .loss/FRt4/detection_loss‚2.281179890284936 1loss/SRqqt3/detection_loss‚0.8514861279389764 2loss/SRqqt3/assignment_loss‚0.9302830391324464 epoch‚7 trainer/global_step‚8599 0loss/FRt1/assignment_loss‚2.3120882702725276 0loss/FBt1/assignment_loss‚0.3881527739285957 0loss/FBt2/assignment_loss‚0.2104809042425142 /loss/FRt3/detection_loss‚1.3549178776937345 1loss/SRqqt2/detection_loss‚2.4225667532413238 /loss/FBt3/detection_loss‚0.8169539699572888 /loss/FRt4/assignment_loss‚2.781644711229536 1loss/FBt3/assignment_loss‚0.11984926721578418 1loss/FBt4/assignment_loss‚0.10966525430871325 .loss/FRt1/detection_loss‚2.980973007423537 .loss/FRt2/detection_loss‚2.310784518798755 2loss/SRqqt1/assignment_loss‚1.2776837163737842 1loss/SRqqt1/detection_loss‚2.3348636167390007 .loss/FBt2/detection_loss‚2.156221214681864 1loss/SRqqt4/detection_loss‚2.0213512517511845 % _timestamp‚1.7885349880841556e+09 _runtime‚10371.61960442 _step‚708 .loss/FBt1/detection_loss‚2.355678152292967 /loss/FBt4/detection_loss‚1.8894296448964338 &loss/total_loss‚38.559273754804444 0loss/FRt2/assignment_loss‚2.9938911980106715 2loss/SRqqt2/assignment_loss‚1.1669205506482432ÄM0Ÿ×xª%‚(:í ÔÁëÔÍâ·à! disk.sdb.out‚1618.39892578125 disk.sda.in‚ 128.6328125 disk.sda1.in‚0 disk.rbd2.out‚ 3.025390625! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.08266626414404! disk.sda.out‚1618.39892578125 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.23000705798982 disk.md0.out‚ 1576.94921875 disk.nvme1n1.in‚ 129.50390625' proc.memory.availableMB‚ 1311.859375 network.sent‚ 2.291129e+07 disk.rbd0.out‚ 3.087890625! disk.nvme0n1.out‚ 128646.15625 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375! disk.nvme0n1.in‚ 7177.58984375* proc.memory.percent‚18.171024322509766$ disk./.usageGB‚2721.084671020508 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 52353.9921875 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0$ memory_percent‚91.99304580688477 disk.dm-0.in‚ 7307.09375" disk.sda2.out‚1618.39892578125" disk.sdb2.out‚1618.39892578125 disk.md0p1.out‚ 1576.94921875 disk.sdb1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚ 3.41101e+06 disk.dm-0.out‚181000.1484375Á…ܽ’«%‚(:‡ ÔÁëÔ—¥Žá gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëžDˆ”¬%‚(:‰ ãÁëÔœŒéà gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y!œ,­%‚(:ô ãÁëÔòÖ®á network.recv‚ 3.416209e+06 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1578.51953125 disk.sdb1.in‚0 disk.sda1.in‚0$ memory_percent‚91.99309349060059 proc.cpu.threads‚17" disk.sda2.out‚1620.03466796875 network.sent‚ 2.3007795e+07 disk.nvme1n1.in‚ 129.50390625 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.md0.out‚ 1578.51953125 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375$ disk./.usageGB‚2721.151782989502( proc.memory.availableMB‚ 1311.8515625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.08454274288565 disk.sda.in‚ 128.6328125! disk.sda.out‚1620.03466796875# disk.nvme1n1.out‚52354.50390625 disk.sda2.in‚ 128.6328125 cpu‚25.22971925142406 disk.sda1.out‚0 disk.rbd2.out‚ 3.025390625! disk.nvme0n1.in‚ 7177.58984375# disk.nvme0n1.out‚128816.8515625 disk.dm-0.in‚ 7307.09375! disk.sdb.out‚1620.03466796875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 199.59375! disk.dm-0.out‚181171.35546875" disk.sdb2.out‚1620.03466796875›93~’®%‚(:‡ òÁëԌּà gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ΰÿt¯%‚(:é òÁëԯʢá disk.dm-0.out‚ 181403.65625 network.recv‚ 3.419726e+06 disk.rbd2.out‚ 3.025390625 disk.md0.in‚ 199.59375 disk.md0.out‚ 1580.0078125$ memory_percent‚91.99309349060059 proc.cpu.threads‚17 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 129.50390625( proc.memory.availableMB‚ 1311.8515625$ disk./.usageGB‚2721.277057647705# disk.nvme1n1.out‚52354.62109375 disk.md0p1.in‚ 199.59375 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766 cpu‚25.229413415707313 disk.sdb.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0! disk.nvme0n1.in‚ 7177.58984375 disk.sda.in‚ 128.6328125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.out‚1621.576171875 disk.dm-0.in‚ 7307.09375 disk.sdb.out‚1621.576171875 disk.sda.out‚1621.576171875 network.sent‚ 2.3011257e+07) disk./.usagePercent‚76.08804547476053 disk.sdb2.out‚1621.576171875 disk.sdb1.in‚0! proc.memory.rssMB‚ 2977.140625$ disk.nvme0n1.out‚129049.03515625 disk.sda2.in‚ 128.6328125 disk.md0p1.out‚ 1580.0078125&Öu’°%‚(:‡ ÂëÔ¶ÎÕÞ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>­®Yt±%‚(:é ÂëÔžÝÍà disk.sda1.out‚0) disk./.usagePercent‚76.09154308692987 disk.sda2.out‚1623.287109375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375* proc.memory.percent‚18.171024322509766! disk.nvme0n1.in‚ 7177.58984375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚1623.287109375 disk.nvme1n1.in‚ 129.50390625! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.424337e+06# disk.nvme0n1.out‚129221.7421875 disk.sda2.in‚ 128.6328125 disk.sda.out‚1623.287109375 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.md0.out‚ 1581.65234375 disk.nvme1n1.out‚ 52354.9375 disk.md0p1.in‚ 199.59375 proc.cpu.threads‚17 network.sent‚ 2.3019627e+07 disk.sdb.out‚1623.287109375 disk.rbd0.in‚0$ memory_percent‚91.99497699737549 disk.dm-0.in‚ 7307.09375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1581.65234375 disk.sdb1.in‚0 disk.rbd2.out‚ 3.025390625 cpu‚25.22913527814523% disk./.usageGB‚2721.4021492004395 disk.md0.in‚ 199.59375 disk.dm-0.out‚181576.6796875) proc.memory.availableMB‚ 1311.54296875tø4²%/ _wandbruntime‚10431  _runtime‚10431ºÈ'øȳ% _step‚709 & lr-AdamW/pg1‚0.0001662046464674939 & lr-AdamW/pg2‚0.0001662046464674939 trainer/global_step‚8649 % _timestamp‚1.7885350478195307e+09 _runtime‚10431.355106902J¯ŠÍ´%Ç & lr-AdamW/pg1‚0.0001662046464674939 & lr-AdamW/pg2‚0.0001662046464674939 trainer/global_step‚8649 % _timestamp‚1.7885350478195307e+09 _runtime‚10431.355106902 _step‚709ÅN(:4µ%/ _wandbruntime‚10431  _runtime‚10431„ÿy(¶%ûO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 _step‚710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715644683595895 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181219710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815645282722304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161163195 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚739877819941888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚729957991167488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400504117 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739876891967488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161163803 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219284407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52577661 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52577055 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚715643758581531 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739877819941888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181219708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885350478205307e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729958009170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705761939807568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400477771 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357704864768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729958009170432 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108586140 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚805287325135360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9919810771456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9919810771456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚729957991167488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181219708 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805287349075456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219258063 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚10431.356118186 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10357933646848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 2192844074¤»ÉÔ·%€P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9918900800000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815645030000128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚739876891967488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181219710 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚8649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9918900800000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚705761956846928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9882726748967 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108586142 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚729957991167488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181219708 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805287349075456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219258063 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚10431.356118186 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10357933646848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219284407 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 _step‚710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715644683595895 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181219710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815645282722304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161163195 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚739877819941888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚729957991167488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400504117 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739876891967488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161163803 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219284407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52577661 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52577055 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚715643758581531 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739877819941888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181219708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885350478205307e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729958009170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705761939807568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400477771 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357704864768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729958009170432 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108586140 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚805287325135360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9919810771456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9919810771456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730039628256256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181239870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219309264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10359104088064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739959632513536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 % _timestamp‚1.7885350489873576e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9920932244992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚711 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚805377550839808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883844022453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400549134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚715723893449353 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161180906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52582701 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9920022260224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181239868 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400549134 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚815736678867968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805377574779904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚739960560501248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚815736432450560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705840974441472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9882936047241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚705840957402112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219309264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9920022260224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219282894 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181239870 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219282894 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚705840957402112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219309264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9920022260224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219282894 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181239870 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219282894 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730039628256256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181239870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219309264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10359104088064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739959632513536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 % _timestamp‚1.7885350489873576e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9920932244992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚711 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚805377550839808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883844022453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400549134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚715723893449353 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161180906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52582701 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9920022260224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181239868 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400549134 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚815736678867968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805377574779904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚739960560501248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚815736432450560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705840974441472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9882936047241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744013933683712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9975898196480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220529309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220502965 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚734038963461632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚734038945458688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10416271035392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚744014861658112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9975898196480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162053633 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚809798634297856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10416493526016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709712819534768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚10488.894786368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402757019 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734038963461632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9937696029071 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182227708 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719650515563839 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9974988225024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚809798610357760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182227710 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109189390 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709712836574128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9974988225024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52864864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820214881393152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚714 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182227710 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162054256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10416493526016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709712819534768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚10488.894786368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402757019 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734038963461632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9937696029071 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182227708 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719650515563839 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9974988225024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚809798610357760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182227710 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109189390 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709712836574128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9974988225024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52864864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820214881393152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚714 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182227710 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162054256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744013933683712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9975898196480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220529309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220502965 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152Ê.Žl4Ì%/ _wandbruntime‚10489  _runtime‚10489¿mÇ1Y"Í%„N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9977019670016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734120564544512 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162072057 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9977019670016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52869983 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚10489.874592526 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182247868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182247868 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220527796 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚809888836062208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10417434314752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744096674229760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _step‚715 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109201457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182247870 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734120582547456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220554166 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162071438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709791854168672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402802036 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109201455 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820306270376960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182247870 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9939721277561 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9938813302349 $ _timestamp‚1.788535106338529e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719730650431661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220527796 WCDeviceStatsMonitor.onOfM±_train_batch_end/allocated_bytes.all.allocated‚744097602217472 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8699 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368HÊô¿'Î%‰N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚734120582547456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402775664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402802036 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109201455 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820306270376960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182247870 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9939721277561 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9938813302349 $ _timestamp‚1.788535106338529e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719730650431661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220527796 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744097602217472 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8699 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9977019670016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734120564544512 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162072057 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9977019670016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52869983 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚10489.874592526 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182247868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182247868 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220527796 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚809888836062208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10417434314752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744096674229760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _step‚715 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109201457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182247870 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734120582547456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220554166 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162071438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709791854168672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748150975399936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814309895580160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7885351581068506e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53153773 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814309919520256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚723658197560511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748150975399936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713663716301328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚8749 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚738119899749888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10031985621504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183235710 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162945820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53153180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183235708 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚738119917752832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748151903374336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221774211 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9993573284179 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221774211 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚738119917752832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109792642 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10031075650048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713663699261968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10541.642452335G$‰Ð Ý%„P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10541.642452335 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183235710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183235708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221747867 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109792640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚738119899749888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748150975399936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814309895580160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7885351581068506e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53153773 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814309919520256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚723658197560511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748150975399936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713663716301328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/gÞd‘:global_step‚8749 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚738119899749888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10031985621504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183235710 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162945820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53153180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183235708 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚738119917752832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748151903374336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221774211 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9993573284179 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221774211 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚738119917752832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109792642 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10031075650048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713663699261968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10033107095040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183255868 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221799068 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53158937 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚713742716856512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10033107095040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚814400145224704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748234643933696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183255868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221799068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚824876237537280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚814400121284608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748233715945984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 405054938 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10542.58667666 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚738201536838656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 405028566 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚824875999508480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183255870 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚723738332428541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚719 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10032197110272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221772698 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 405028566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10032197110272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405054938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53159542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183255870 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚723737407413969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162963642 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚713742733895872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚738201536838656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚738201518835712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10475878223872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885351590504577e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748234643933696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ý><'à%N E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10476092312576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10033107095040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183255868 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221799068 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53158937 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚713742716856512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10033107095040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚814400145224704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748234643933696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183255868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221799068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚824876237537280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚814400121284608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748233715945984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 405054938 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10542.58667666 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚738201536838656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 405028566 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚824875999508480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183255870 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚723738332428541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚719 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10032197110272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221772698 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 405028566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10032197110272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405054938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53159542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183255870 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚723737407413969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162963642 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚713742733895872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚738201536838656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚738201518835712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10475878223872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885351590504577e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748234643933696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572Ϫ9’4á%/ _wandbruntime‚10542  _runtime‚10542ïxÏË^â%Ø % _timestamp‚1.7885351590523407e+09 0loss/FRt4/assignment_loss‚2.6173114577929177 1loss/SRqqt4/assignment_loss‚1.166180175252077 _runtime‚10542.587646815 2loss/SRqqt1/assignment_loss‚1.2585860853298352 1loss/FBt4/assignment_loss‚0.14223287334945295 .loss/FRt2/detection_loss‚2.104813238790632 /loss/FRt3/detection_loss‚1.1534064303051277 trainer/global_step‚8749 /loss/FRt2/assignment_loss‚3.673501422238904 1loss/FBt3/assignment_loss‚0.03844149917821428 /loss/FBt1/detection_loss‚2.1947305272607243 /loss/FBt4/detection_loss‚1.9257758311603381 _step‚720 %loss/total_loss‚38.29067558451281 0loss/FRt1/assignment_loss‚3.1168733931356862 0loss/FBt1/assignment_loss‚0.3015081571778704 1loss/SRqqt2/detection_loss‚2.0071482621133327 .loss/FBt2/detection_loss‚2.265607334318615 /loss/FBt3/detection_loss‚0.6392854103973756 2loss/SRqqt3/assignment_loss‚0.8403746845456315 epoch‚7 1loss/SRqqt1/detection_loss‚2.2399952778449426 0loss/FRt3/assignment_loss‚1.3711743031639652 1loss/SRqqt2/assignment_loss‚0.995393369696103 1loss/FBt2/assignment_loss‚0.07387349420050764 /loss/FRt1/detection_loss‚2.8724148811832553 /loss/FRt4/detection_loss‚2.4621416227685082 1loss/SRqqt3/detection_loss‚0.8248550923650756 1loss/SRqqt4/detection_loss‚2.0050507609437154û=Kcã%Ý /loss/FRt2/assignment_loss‚3.673501422238904 1loss/FBt3/assignment_loss‚0.03844149917821428 /loss/FBt1/detection_loss‚2.1947305272607243 /loss/FBt4/detection_loss‚1.9257758311603381 _step‚720 %loss/total_loss‚38.29067558451281 0loss/FRt1/assignment_loss‚3.1168733931356862 0loss/FBt1/assignment_loss‚0.3015081571778704 1loss/SRqqt2/detection_loss‚2.0071482621133327 .loss/FBt2/detection_loss‚2.265607334318615 /loss/FBt3/detection_loss‚0.6392854103973756 2loss/SRqqt3/assignment_loss‚0.8403746845456315 epoch‚7 1loss/SRqqt1/detection_loss‚2.2399952778449426 0loss/FRt3/assignment_loss‚1.3711743031639652 1loss/SRqqt2/assignment_loss‚0.995393369696103 1loss/FBt2/assignment_loss‚0.07387349420050764 /loss/FRt1/detection_loss‚2.8724148811832553 /loss/FRt4/detection_loss‚2.4621416227685082 1loss/SRqqt3/detection_loss‚0.8248550923650756 1loss/SRqqt4/detection_loss‚2.0050507609437154 % _timestamp‚1.7885351590523407e+09 0loss/FRt4/assignment_loss‚2.6173114577929177 1loss/SRqqt4/assignment_loss‚1.166180175252077 _runtime‚10542.587646815 2loss/SRqqt1/assignment_loss‚1.2585860853298352 1loss/FBt4/assignment_loss‚0.14223287334945295 .loss/FRt2/detection_loss‚2.104813238790632 /loss/FRt3/detection_loss‚1.1534064303051277 trainer/global_step‚8749Ð<§óÚ“ä%‚(:ˆ ùÂëÔ»Òá gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è$Gzå%‚(:ï ùÂëÔŸÖ¨á disk.dm-0.in‚ 8010.609375 disk.sdb.in‚ 70.9609375 disk.sdb.out‚1636.2236328125# disk.nvme1n1.out‚52540.55859375( disk./.usagePercent‚76.0596716400055 disk.sdb1.out‚0 disk.sda1.in‚0! disk.sda2.out‚1636.2236328125% disk./.usageGB‚2720.2622718811035 disk.rbd0.out‚ 3.087890625 disk.nvme0n1.in‚ 7515.9609375 disk.md0.out‚ 1594.12890625 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 494.6484375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚130109.18359375 disk.md0.in‚ 199.59375 disk.sda.out‚1636.2236328125 proc.cpu.threads‚17 network.sent‚ 2.3280026e+07 disk.rbd2.out‚ 3.025390625! disk.sdb2.out‚1636.2236328125 disk.md0p1.out‚ 1594.12890625 disk.sda2.in‚ 128.6328125 disk.dm-0.out‚182649.7421875* proc.memory.percent‚18.171024322509766 disk.md0p1.in‚ 199.59375$ memory_percent‚91.99512004852295) proc.memory.availableMB‚ 1311.51953125 cpu‚25.226672054807626 disk.sdb2.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.4597e+06‘\þ·“æ%‚(:ˆ ˆÃëÔš™Ùà gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó«>†ç%‚(:÷ ˆÃëÔ©åüà" disk.sdb2.out‚1637.72509765625 network.sent‚ 2.3377443e+07 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚130109.23046875 disk.md0.out‚ 1595.57421875 disk.sda.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 network.recv‚ 3.465171e+06! disk.sda.out‚1637.72509765625 cpu‚25.226399827449164 disk.sda1.in‚0 disk.nvme0n1.in‚ 7535.1484375 disk.dm-0.in‚ 8050.80859375! disk.sdb.out‚1637.72509765625$ memory_percent‚91.99378490447998! proc.memory.rssMB‚ 2977.140625 disk.sdb1.in‚0 disk.rbd2.out‚ 3.025390625 disk.sda2.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375) disk./.usagePercent‚76.05966140059444$ disk./.usageGB‚2720.261905670166 disk.md0p1.out‚ 1595.57421875" disk.sda2.out‚1637.72509765625 disk.md0.in‚ 199.59375 disk.­4Îpódm-0.out‚ 182764.765625) proc.memory.availableMB‚ 1311.73828125 disk.nvme1n1.in‚ 515.66015625 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0# disk.nvme1n1.out‚52655.53515625 disk.md0p1.in‚ 199.59375 proc.cpu.threads‚17âÙÉb’è%‚(:‡ —ÃëÔ©µŸà gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[[ç{é%‚(:ð —ÃëÔªŒ›á disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 516.40234375# disk.nvme1n1.out‚52793.85546875! proc.memory.rssMB‚ 2977.140625 cpu‚25.226078201403947 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.md0p1.out‚ 1597.37109375 disk.sdb1.in‚0% disk./.usageGB‚2720.3980102539062 disk.md0.out‚ 1597.37109375 disk.sdb2.out‚1639.576171875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7535.9296875$ disk.nvme0n1.out‚130179.25390625$ memory_percent‚91.99397563934326) proc.memory.availableMB‚ 1311.70703125) disk./.usagePercent‚76.06346694172001 disk.sdb.out‚1639.576171875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 network.recv‚ 3.469206e+06 disk.sda2.out‚1639.576171875 disk.dm-0.out‚ 182973.109375 disk.sdb2.in‚ 70.9609375 disk.sda.out‚1639.576171875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0 disk.rbd2.out‚ 3.025390625 network.sent‚ 2.3382896e+07 disk.dm-0.in‚ 8052.33203125 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0¢Äô“ê%‚(:ˆ ¦ÃëÔò™¬à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M.ßí~ë%‚(:ó ¦ÃëÔàÖ—á disk.sda1.out‚0 disk.rbd2.out‚ 3.025390625" disk.sdb2.out‚1641.12451171875! disk.nvme1n1.out‚ 52882.046875$ memory_percent‚91.99397563934326 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7535.9296875$ disk.nvme0n1.out‚130248.84765625 disk.nvme1n1.in‚ 516.40234375 disk.md0p1.out‚ 1598.8515625 disk.rbd0.out‚ 3.087890625( disk./.usagePercent‚76.0547698419308 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.225805925576054 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 disk.sdb1.out‚0 network.sent‚ 2.338851e+07! disk.sdb.out‚1641.12451171875 disk.sda.in‚ 128.6328125 disk.dm-0.in‚ 8052.33203125 disk.sdb.in‚ 70.9609375) proc.memory.availableMB‚ 1311.70703125* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 network.recv‚ 3.472939e+06$ disk./.usageGB‚2720.086959838867 disk.sda1.in‚0" disk.sda2.out‚1641.12451171875 disk.md0.out‚ 1598.8515625! disk.dm-0.out‚183130.89453125! disk.sda.out‚1641.12451171875\»‘4ì%/ _wandbruntime‚10595  _runtime‚10595³ruÊí%Ä % _timestamp‚1.7885352123743637e+09 _runtime‚10595.910037579 _step‚721 ' lr-AdamW/pg1‚0.00015311761706222933 ' lr-AdamW/pg2‚0.00015311761706222933 trainer/global_step‚8799•c=Ïî%É ' lr-AdamW/pg2‚0.00015311761706222933 trainer/global_step‚8799 % _timestamp‚1.7885352123743637e+09 _runtime‚10595.910037579 _step‚721 ' lr-AdamW/pg1‚0.00015311761706222933Ѩ ‰4ï%/ _wandbruntime‚10595  _runtime‚10595'ñg(ð%P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚752288945090560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742200854041088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184243708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223019113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752288017116160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚752288017116160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53441392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742200872044032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885352123759894e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110395890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚10595.911849208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223019113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829354392879104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10050358514291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10049450539287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742200854041088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚829354639309824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222992769 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742200872044032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10088073046528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110395892 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10088073046528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727664954542819 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚8799 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184243710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184243708 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717614596028528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚722 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752288945090560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53442014 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10533212076544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163837282 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 407236477 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚818821204742656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10087163075072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717614578989168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222992769 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10087163075072wã (ñ%†P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184243710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184243708 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717614596028528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚722 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752288945090560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53442014 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10533212076544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163837282 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 407236477 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚818821204742656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10087163075072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717614578989168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222992769 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10087163075072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚752288945090560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742200854041088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184243708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223019113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752288017116160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚752288017116160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53441392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742200872044032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885352123759894e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110395890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚10595.911849208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223019113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829354392879104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10050358514291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10049450539287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742200854041088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚829354639309824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222992769 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742200872044032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10088073046528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110395892 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10088073046528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727664954542819 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚8799Ò‚ØÒ¸4ò%/ _wandbruntime‚10597  _runtime‚10597ñnøL ó%‹N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 407281468 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _runtime‚10597.123919011 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717693613623072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407281468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10089194520064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10534613970432 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407307840 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110407957 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10534397784576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885352135869615e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10050567812565 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163855796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53447212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10089194520064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚829445804291584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223017600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚752371685649920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10088284535296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚818911430447104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚752370757662208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163855167 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚829446044417536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚723 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223017600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184263868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184263868 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10088284535296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚752371685649920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110407955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53447839 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_tN'öÅrain_batch_end/requested_bytes.all.current‚ 925014572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚742282491129856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727744164396277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742282473126912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10051475787777 trainer/global_step‚8799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 184263870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223043970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407307840 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407281468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10089194520064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10534613970432 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407307840 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110407957 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10534397784576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885352135869615e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10050567812565 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163855796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53447212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10089194520064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚829445804291584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223017600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚752371685649920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10088284535296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚818911430447104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚752370757662208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163855167 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚829446044417536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚723 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223017600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184263868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184263868 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10088284535296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚752371685649920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110407955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53447839 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚742282491129856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727744164396277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742282473126912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10051475787777 trainer/global_step‚8799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 184263870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223043970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464Ó¸ù£4õ%/ _wandbruntime‚10597  _runtime‚10597êÔ“3Yö%Ó 1loss/SRqqt4/assignment_loss‚1.759776611266465 .loss/FBt1/detection_loss‚2.957841846677992 0loss/FBt1/assignment_loss‚0.3658581423955588 0loss/FBt3/detection_loss‚0.44589718290704433 % _timestamp‚1.7885352135898085e+09 2loss/SRqqt2/assignment_loss‚1.3666662308573723 2loss/SRqqt3/assignment_loss‚0.8133857414531694 1loss/FBt2/assignment_loss‚0.16184539617097396 /loss/FBt3/assignment_loss‚0.101660117174362 0loss/FBt4/assignment_loss‚0.4932053624667798 .loss/FRt2/detection_loss‚2.318795323346069 /loss/FBt2/detection_loss‚1.6569239905006008 0loss/FRt2/assignment_loss‚2.9201839854640346 0loss/FRt4/assignment_loss‚2.9324049392472142 .loss/FRt1/detection_loss‚3.094132295846939 1loss/SRqqt2/detection_loss‚2.1978838992118837 1loss/SRqqt3/detection_loss‚0.8477376859497141 epoch‚7 /loss/FRt3/detection_loss‚1.5279844202048762 .loss/FRt4/detection_loss‚2.314128159828808 /loss/SRqqt1/detection_loss‚2.34419832969534 1loss/SRqqt4/detection_loss‚2.4933073726193657 trainer/global_step‚8799 _runtime‚10597.125137607 0loss/FRt1/assignment_loss‚2.9190550327301024 0loss/FRt3/assignment_loss‚1.0670068809683895 2loss/SRqqt1/assignment_loss‚1.3712209425866604 .loss/FBt4/detection_loss‚2.050438914448023 _step‚724 %loss/total_loss‚40.52153880401774j¦0^÷%Ø 2loss/SRqqt2/assignment_loss‚1.3666662308573723 2loss/SRqqt3/assignment_loss‚0.8133857414531694 1loss/FBt2/assignment_loss‚0.16184539617097396 /loss/FBt3/assignment_loss‚0.101660117174362 0loss/FBt4/assignment_loss‚0.4932053624667798 .loss/FRt2/detection_loss‚2.318795323346069 /loss/FBt2/detection_loss‚1.6569239905006008 0loss/FRt2/assignment_loss‚2.9201839854640346 0loss/FRt4/assignment_loss‚2.9324049392472142 .loss/FRt1/detection_loss‚3.094132295846939 1loss/SRqqt2/detection_loss‚2.1978838992118837 1loss/SRqqt3/detection_loss‚0.8477376859497141 epoch‚7 /loss/FRt3/detection_loss‚1.5279844202048762 .loss/FRt4/detection_loss‚2.314128159828808 /loss/SRqqt1/detection_loss‚2.34419832969534 1loss/SRqqt4/detection_loss‚2.4933073726193657 trainer/global_step‚8799 _runtime‚10597.125137607 0loss/FRt1/assignment_loss‚2.9190550327301024 0loss/FRt3/assignment_loss‚1.0670068809683895 2loss/SRqqt1/assignment_loss‚1.3712209425866604 .loss/FBt4/detection_loss‚2.050438914448023 _step‚724 %loss/total_loss‚40.52153880401774 1loss/SRqqt4/assignment_loss‚1.759776611266465 .loss/FBt1/detection_loss‚2.957841846677992 0loss/FBt1/assignment_loss‚0.3658581423955588 0loss/FBt3/detection_loss‚0.44589718290704433 % _timestamp‚1.7885352135898085e+09Ô†}Œ"“ø%‚(:ˆ µÃëÔý¶îà gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00áŒyù%‚(:î µÃëÔÀÜâ disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 3.025390625" disk.sda2.out‚1642.57080078125 proc.cpu.threads‚17 network.recv‚ 3.477461e+06$ disk./.usageGB‚2720.187370300293 disk.md0.out‚ 1600.234375 disk.sda.in‚ 128.6328125! disk.sdb.out‚1642.57080078125" disk.sdb2.out‚1642.57080078125 disk.rbd0.out‚ 3.087890625! proc.memory.rssMB‚ 2977.140625 disk.md0p1.out‚ 1600.234375! disk.sda.out‚1642.57080078125 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 8052.33203125 disk.sda1.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 1311.703125* proc.memory.percent‚18.171024322509766 disk.nvme1n1.in‚ 516.40234375$ memory_percent‚91.99399948120117 network.sent‚ 2.3405352e+07( disk./.usagePercent‚76.0575773604573 disk.nvme0n1.in‚ 7535.9296875# disk.nvme0n1.out‚130253.5546875 disk.dm-0.out‚183174.6953125! disk.nvme1n1.out‚ 52921.140625 disk.rbd0.in‚0 cpu‚25.225526661243293 Gž3”ú%‚(:‰ ÄÃëÔò›‘à gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mŸvmû%‚(:â ÄÃëÔ°¸Üà disk.rbd2.out‚ 3.025390625 disk.dm-0.in‚ 8053.76171875 disk.nvme1n1.in‚ 517.1640625 disk.sdb1.in‚0" disk.sda2.out‚1644.15576171875 disk.md0.out‚1601.75! disk.sda.out‚1644.15576171875 disk.md0p1.out‚1601.75 network.recv‚ 3.483187e+06 disk.nvme0n1.out‚ 130448.25 disk.dm-0.out‚183369.9765625" disk.sdb2.out‚1644.15576171875$ memory_percent‚91.99399948120117 network.sent‚ 2.3502708e+07 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125' proc.memory.availableMB‚ 1311.703125* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.06352624497586 disk.sda1.in‚0 disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7536.59765625 disk.rbd0.out‚ 3.087890625 proc.cpu.threads‚17! disk.sdb.out‚1644.15576171875" disk.nvme1n1.out‚ 52921.7265625 cpu‚25.22524220706119$ disk./.usageGB‚2720.400131225586 disk.sdb1.out‚0 disk.rbd2.in‚0æeq“ü%‚(:ˆ ÓÃëÔÈúÏà gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³´aÓsý%‚(:è ÓÃëÔüÇá disk.md0p1.in‚ 199.59375 disk.md0.out‚ 1603.53125 disk.dm-0.in‚ 8055.8515625! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 518.0234375 disk.rbd0.in‚0" disk./.usageGB‚2720.7138671875 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1603.53125* proc.memory.percent‚18.171024322509766 cpu‚25.224968038743896 network.sent‚ 2.3505828e+07! disk.sda.out‚1646.00244140625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7537.828125 disk.dm-0.out‚183474.8203125" disk.sdb2.out‚1646.00244140625! disk.nvme1n1.out‚ 52921.984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚130552.8359375$ memory_percent‚91.99399948120117' proc.memory.availableMB‚ 1311.703125 network.recv‚ 3.486782e+06 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0" disk.sda2.out‚1646.00244140625 disk.md0.in‚ 199.59375) disk./.usagePercent‚76.07229843377968 disk.rbd2.out‚ 3.025390625 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1646.00244140625 disk.sdb2.in‚ 70.9609375 disk.sdb.in‚ 70.9609375ÇוВþ%‚(:‡ âÃëÔÂÄ£à gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Ñ¬€ÿ%‚(:õ âÃëÔõÆ”á! disk.sda.out‚1647.61962890625 disk.md0p1.in‚ 199.59375* proc.memory.percent‚18.171024322509766 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1647.61962890625" disk.sdb2.out‚1647.61962890625 disk.md0p1.out‚ 1605.0859375) proc.memory.availableMB‚ 1311.90234375# disk.nvme1n1.out‚52922.11328125 disk.rbd0.in‚0 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1647.61962890625 cpu‚25.224724133942125) disk./.usagePercent‚76.07752458653259 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.828125! disk.dm-0.out‚183670.40234375$ disk./.usageGB‚2720.900779724121 disk.rbd2.out‚ 3.025390625 disk.md0.out‚ 1605.0859375$ memory_percent‚91.99278354644775 network.recv‚ 3.490799e+06 disk.sda2.in‚ 128.6328125 disk.md0.in‚ 199.59375 disk.nvme1n1.in‚ 518.03515625! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17# disk.nvme0n1.out‚130748.2890625 disk.dm-0.in‚ 8055.86328125 network.sent‚ 2.3513741e+07 disk.sdb1.in‚0 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.63281256Þ4€&/ _wandbruntime‚10656  _runtime‚10656ŠÉœÕÊ&Ä ' lr-AdamW/pg2‚0.00014884275770536896 trainer/global_step‚8849 % _timestamp‚1.7885352732648542e+09 _runtime‚10656.800598939 _step‚725 ' lr-AdamW/pg1‚0.00014884275770536896ŒƒTÏ‚&É _step‚725 ' lr-AdamW/pg1‚0.00014884275770536896 ' lr-AdamW/pg2‚0.00014884275770536896 trainer/global_step‚8849 % _timestamp‚1.7885352732648542e+09 _runtime‚10656.800598939ÕÂÌ+™4ƒ&/ _wandbruntime‚10656  _runtime‚10656£V56(„&P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚756425986806784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚756425058832384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110999140 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409489379 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185251710 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885352732662048e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10591773478912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10105327794395 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚833924504808960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚731671711525127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10106235769399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚823332466024960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833924239503872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53729732 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721565458716368 trainer/global_step‚8849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185251708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224264015 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10144160471552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756425058832384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164728872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823332489965056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731670786510763 _runtime‚10656.802916479 _step‚726 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10143250500096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110999142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746281808332288 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731670786510763 _runtime‚10656.802916479 _step‚726 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10143250500096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.larglAW«„#e_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110999142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746281808332288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚823332466024960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833924239503872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53729732 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721565458716368 trainer/global_step‚8849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185251708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224264015 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10144160471552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756425058832384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164728872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823332489965056 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823422691729408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721644476310912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10145281945088 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111011205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10144371960320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185271870 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185271870 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10107353042885 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885352744777505e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10592951026688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164746708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224262502 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185271868 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚756508727366144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834015891270656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409560742 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224262502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721644493350272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224288872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746363427418112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚756507799378432 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224288872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚10658.014957994 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409534370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10593175601152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731750921378585 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10106445067673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185271868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚746363427418112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚756508727366144܉ 0'ˆ&N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746363445421056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164746708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224262502 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185271868 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚756508727366144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834015891270656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409560742 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224262502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721644493350272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224288872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746363427418112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚756507799378432 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224288872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚10658.014957994 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409534370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10593175601152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731750921378585 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10106445067673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185271868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚746363427418112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚756508727366144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10144371960320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823422691729408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721644476310912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10145281945088 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111011205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10144371960320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185271870 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185271870 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10107353042885 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885352744777505e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10592951026688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186259708 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚760563028523008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54017629 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10162113024507 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200247896576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10199337925120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186259710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735678468507435 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725516338443568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225508917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111602390 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10161205049503 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788535333675409e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225482573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750362780626432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚10717.211744181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10650274524160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚760562100548608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186259708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735677543493071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165620653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10650499111936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225508917 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186259710 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚760562100548608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10200247896576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10199337925120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165620019 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚730 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346\pJÒ (™&…P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225508917 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186259710 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚760562100548608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10200247896576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10199337925120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165620019 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚730 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750362762623488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54018261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚838494274299392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚760563028523008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚750362762623488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186259708 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚760563028523008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54017629 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10162113024507 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200247896576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10199337925120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186259710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735678468507435 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725516338443568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225508917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111602390 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10161205049503 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788535333675409e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225482573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750362780626432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚10717.211744181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10650274524160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚760562100548608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186259708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735677543493071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165620653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10650499111936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640Ú'J¨74š&/ _wandbruntime‚10718  _runtime‚10718ù¸OÄ)›&‹N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750444381709312 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111614457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885353348803165e+09 _step‚731 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411787272 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10718.417952205 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750444381709312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚760644841094656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54023378 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111614455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186279870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225533774 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165637833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54024004 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10200459385344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚735758603375465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760645769082368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10200459385344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411813644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165638461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225533774 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225507404 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725595373077472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750444399712256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 trainer/global_step‚8899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186279870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10651434285056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750444399712256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10718.417952205 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750444381709312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚760644841094656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54023378 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111614455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186279870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225533774 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165637833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54024004 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10200459385344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚735758603375465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760645769082368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10200459385344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411813644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165638461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225533774 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225507404 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725595373077472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750444399712256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 trainer/global_step‚8899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186279870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10651434285056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750444399712256 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768Ûšƒ,4&/ _wandbruntime‚10718  _runtime‚10718“ÿ®I_ž&Ù 2loss/SRqqt4/assignment_loss‚1.3993921405635774 &loss/total_loss‚39.966772529472706 epoch‚7 2loss/SRqqt3/assignment_loss‚0.9002301583711344 1loss/FBt4/assignment_loss‚0.24155826271937889 1loss/SRqqt3/detection_loss‚0.8996578590268944 0loss/FRt2/assignment_loss‚3.1765494817181636 0loss/FRt4/assignment_loss‚2.7576602243858837 /loss/FRt2/detection_loss‚2.3085455026645687 /loss/FBt1/detection_loss‚2.5774124369901767 2loss/FBt1/assignment_loss‚0.027148834507443098 .loss/FRt4/detection_loss‚2.140074241663451 0loss/SRqqt1/detection_loss‚2.375146700276269 /loss/FRt1/detection_loss‚3.0973032812277475 .loss/FRt3/detection_loss‚1.206379739706641 1loss/SRqqt2/detection_loss‚2.1690467217484035 /loss/SRqqt4/detection_loss‚2.11417232779786 /loss/FBt2/detection_loss‚2.0072743025693027 /loss/FBt4/detection_loss‚2.2467000759564915 _step‚732 0loss/FRt3/assignment_loss‚0.8085686075153635 2loss/SRqqt1/assignment_loss‚1.7044134277988363 2loss/SRqqt2/assignment_loss‚1.5498996919675454 1loss/FBt3/assignment_loss‚0.10220724120340648 /loss/FRt1/assignment_loss‚3.314813400308291 1loss/FBt2/assignment_loss‚0.41331115690685105 /loss/FBt3/detection_loss‚0.4293067118790197 trainer/global_step‚8899 % _timestamp‚1.7885353348837175e+09 _runtime‚10718.41914192Þõ«ÌdŸ&Þ /loss/FBt4/detection_loss‚2.2467000759564915 _step‚732 0loss/FRt3/assignment_loss‚0.8085686075153635 2loss/SRqqt1/assignment_loss‚1.7044134277988363 2loss/SRqqt2/assignment_loss‚1.5498996919675454 1loss/FBt3/assignment_loss‚0.10220724120340648 /loss/FRt1/assignment_loss‚3.314813400308291 1loss/FBt2/assignment_loss‚0.41331115690685105 /loss/FBt3/detection_loss‚0.4293067118790197 trainer/global_step‚8899 % _timestamp‚1.7885353348837175e+09 _runtime‚10718.41914192 2loss/SRqqt4/assignment_loss‚1.3993921405635774 &loss/total_loss‚39.966772529472706 epoch‚7 2loss/SRqqt3/assignment_loss‚0.9002301583711344 1loss/FBt4/assignment_loss‚0.24155826271937889 1loss/SRqqt3/detection_loss‚0.8996578590268944 0loss/FRt2/assignment_loss‚3.1765494817181636 0loss/FRt4/assignment_loss‚2.7576602243858837 /loss/FRt2/detection_loss‚2.3085455026645687 /loss/FBt1/detection_loss‚2.5774124369901767 2loss/FBt1/assignment_loss‚0.027148834507443098 .loss/FRt4/detection_loss‚2.140074241663451 0loss/SRqqt1/detection_loss‚2.375146700276269 /loss/FRt1/detection_loss‚3.0973032812277475 .loss/FRt3/detection_loss‚1.206379739706641 1loss/SRqqt2/detection_loss‚2.1690467217484035 /loss/SRqqt4/detection_loss‚2.11417232779786 /loss/FBt2/detection_loss‚2.0072743025693027Ü™º$‘ &‚(:† ­ÄëÔæŒÛá gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¼]~¡&‚(:ó ­ÄëÔûàŸâ network.recv‚ 3.512826e+06 disk.dm-0.out‚ 184609.453125 disk.sda1.in‚0 disk.nvme0n1.in‚ 7537.84375! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1655.54541015625$ disk./.usageGB‚2721.150318145752 disk.dm-0.in‚ 8055.95703125 disk.rbd2.out‚ 3.025390625 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1655.54541015625 disk.nvme1n1.in‚ 518.11328125 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1655.54541015625 proc.cpu.threads‚17" disk.sdb2.out‚1655.54541015625 disk.sdb1.out‚0 disk.md0.out‚ 1612.68359375 network.sent‚ 2.3642302e+07 disk.sda.in‚ 128.6328125" disk.nvme1n1.out‚ 52923.6953125 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.223360158073135 disk.sda2.in‚ 128.6328125) disk./.usagePercent‚76.08450178524132 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1612.68359375 disk.rbd0.out‚ 3.087890625# disk.nvme0n1.out‚131685.7578125 disk.md0.in‚ 199.59375$ memory_percent‚91.99631214141846) proc.memory.availableMB‚ 1311.32421875mÁ³Ü”¢&‚(:‰ ¼ÄëÔ×ô©à gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n |*~£&‚(:ó ¼ÄëÔ©ôàà network.sent‚ 2.3752597e+07 disk.sda2.in‚ 128.6328125 disk.dm-0.out‚ 184815.703125# disk.nvme0n1.out‚131891.8515625$ memory_percent‚91.99793338775635! disk.sda.out‚1657.72802734375 disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375! proc.memory.rssMB‚ 2977.140625 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sdb1.in‚0* proc.memory.percent‚18.171024322509766 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625% disk./.usageGB‚2721.2146911621094 disk.dm-0.in‚ 8055.95703125" disk.sdb2.out‚1657.72802734375" disk.nvme1n1.out‚ 52923.8515625 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1614.80078125 network.recv‚ 3.518625e+06 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0) proc.memory.availableMB‚ 1311.05859375) disk./.usagePercent‚76.08630168172088 disk.md0.in‚ 199.59375 disk.md0.out‚ 1614.80078125! disk.sdb.out‚1657.72802734375 cpu‚25.22310259710627 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 518.11328125 disk.sdb1.out‚0 proc.cpu.threads‚17" disk.sda2.out‚1657.72802734375 disk.rbd0.in‚0¬kZö”¤&‚(:‰ ËÄëÔ½µšà gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¥úYt¥&‚(:é ËÄëÔȯèà disk.rbd2.out‚ 3.025390625 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 8055.95703125! disk.sdb.out‚1659.11376953125* proc.memory.percent‚18.171024322509766 disk.sdb1.in‚0$ memory_percent‚91.99643135070801 cpu‚25.22286140577957 disk.sdb2.in‚ 70.9609375 disk.sda1.in‚0 disk.md0.out‚1616.125 disk.sdb.in‚ 70.9609375( proc.memory.availableMB‚ 1311.3046875) disk./.usagePercent‚76.09154330025093! disk.sda.out‚1659.11376953125 disk.md0p1.in‚ 199.59375" disk.sda2.out‚1659.11376953125 disk.dm-0.out‚184999.8046875 network.sent‚ 2.3758006e+07 network.recv‚ 3.52251e+06 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.out‚ 3.087890625$ disk.nvme0n1.out‚132075.68359375 disk.md0.in‚ 199.59375! proc.memory.rssMB‚ 2977.140625# disk.nvme1n1.out‚52924.12109375 disk.rbd0.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2721.402156829834 disk.sda.in‚ 128.6328125" disk.sdb2.out‚1659.11376953125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚1616.125 disk.sda1.out‚0 ~Ðß’¦&‚(:‡ ÚÄëÔ£à‡á gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.564' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñbð¢|§&‚(:ñ ÚÄëÔÍÐÍá network.sent‚ 2.376151e+07 network.recv‚ 3.526297e+06 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1617.57421875 disk.dm-0.out‚185125.1796875 disk.sdb.out‚1660.6123046875 disk.nvme1n1.in‚ 518.11328125( proc.memory.availableMB‚ 1311.3046875% disk./.usageGB‚2721.4021949768066 disk.rbd2.out‚ 3.025390625 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375 disk.sdb1.out‚0$ memory_percent‚91.99643135070801* proc.memory.percent‚18.171024322509766 cpu‚25.222571030151183 disk.rbd2.in‚0 disk.sda.in‚ 128.6328125# disk.nvme1n1.out‚52924.25390625 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1617.57421875) disk./.usagePercent‚76.09154436685625! disk.sdb2.out‚1660.6123046875 disk.rbd0.in‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚132213.17578125! disk.sda2.out‚1660.6123046875 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 8055.95703125 disk.sda.out‚1660.6123046875 disk.nvme0n1.in‚ 7537.84375 disk.sdb2.in‚ 70.9609375ôƹ¾4¨&/ _wandbruntime‚10777  _runtime‚10777߸8È©& & lr-AdamW/pg1‚0.0001404296234869085 & lr-AdamW/pg2‚0.0001404296234869085 trainer/global_step‚8949 % _timestamp‚1.7885353940221245e+09 _runtime‚10777.557668883 _step‚733Ų́ͪ&Ç & lr-AdamW/pg1‚0.0001404296234869085 & lr-AdamW/pg2‚0.0001404296234869085 trainer/global_step‚8949 % _timestamp‚1.7885353940221245e+09 _runtime‚10777.557668883 _step‚733ÝÔ‹;4«&/ _wandbruntime‚10777  _runtime‚10777ÊBUY$¬&P UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739684300475379 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885353940232563e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10217990279615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187267708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54307376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739685225489743 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226727475 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764699142264832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚734 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚10777.558845868 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10217082304611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166513663 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚729467218170768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187267708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832355060409856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10255425350144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413995183 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10256335321600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10256335321600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226753819 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112205640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚754443716914688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54308021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚754443734917632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚843063640219136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226753819 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754443734917632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚754443716914688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10255425350144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152Å›L (­&†P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚734 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚10777.558845868 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10217082304611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166513663 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚729467218170768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187267708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832355060409856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10255425350144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413995183 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10256335321600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10256335321600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226753819 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112205640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚754443716914688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54308021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚754443734917632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚843063640219136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226753819 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754443734917632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚754443716914688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10255425350144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739684300475379 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885353940232563e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10217990279615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187267708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54307376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739685225489743 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226727475 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764699142264832Þâw³4®&/ _wandbruntime‚10778  _runtime‚10778M3ò‰'¯&ŠN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10256546810368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832445286114304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739765360357773 _step‚735 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187287868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10709783650304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187287868 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754525354003456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 414040174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187287870 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112217707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226752306 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754525354003456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10710010321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10218199577889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764781882810880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10256546810368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚754525336000512 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10778.502314349 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843155045824512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 414040174 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166531509 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739764435343201 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166530854 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10219107553101 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764782810798592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚729546235765312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 414066546 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚729546252804672 trainer/global_step‚8949 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226752306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54313802 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764782810798592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54313149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843155296436224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414066546 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0k(Ð'°&N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 414066546 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚729546252804672 trainer/global_step‚8949 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226752306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54313802 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764782810798592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54313149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843155296436224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414066546 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10256546810368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832445286114304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739765360357773 _step‚735 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187287868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10709783650304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187287868 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754525354003456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 414040174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187287870 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112217707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226752306 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754525354003456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10710010321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10218199577889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764781882810880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10256546810368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚754525336000512 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10778.502314349 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843155045824512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 414040174 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166531509 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739764435343201 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166530854 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10219107553101 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764782810798592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚729546235765312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ßþ%›Z4±&/ _wandbruntime‚10778  _runtime‚10778¢"R\²&Ö /loss/FBt1/detection_loss‚2.8140632033348085 _step‚736 /loss/FRt1/assignment_loss‚2.826131968787222 0loss/FBt1/assignment_loss‚0.1473637068278549 1loss/FBt2/assignment_loss‚0.18474849588925066 0loss/SRqqt2/detection_loss‚2.151746561129888 1loss/SRqqt3/detection_loss‚0.8702193274039705 % _timestamp‚1.7885353949685004e+09 _runtime‚10778.503767885 0loss/SRqqt1/detection_loss‚2.432548201084137 2loss/SRqqt3/assignment_loss‚0.8801425510714322 /loss/FRt2/detection_loss‚2.5226150252550332 /loss/FRt4/detection_loss‚2.1884415289577173 /loss/FBt3/detection_loss‚0.3980803095975951 /loss/FRt4/assignment_loss‚3.099150666168758 2loss/SRqqt4/assignment_loss‚1.0737465342570995 /loss/FRt1/detection_loss‚2.7883351914810413 1loss/SRqqt4/detection_loss‚2.2120416421314766 trainer/global_step‚8949 0loss/FRt3/assignment_loss‚1.1076972461663759 2loss/SRqqt1/assignment_loss‚1.4014532905817032 1loss/SRqqt2/assignment_loss‚1.513195600360632 1loss/FBt3/assignment_loss‚0.08965080899873357 .loss/FBt2/detection_loss‚1.928332057264116 epoch‚7 0loss/FRt2/assignment_loss‚3.7230122813156674 /loss/FBt4/detection_loss‚1.9187948107719421 %loss/total_loss‚39.87223887653407 0loss/FBt4/assignment_loss‚0.1810909263976279 /loss/FRt3/detection_loss‚1.4196369412999887eDÍ%³&Û 0loss/FBt4/assignment_loss‚¯×§±<0.1810909263976279 /loss/FRt3/detection_loss‚1.4196369412999887 /loss/FBt1/detection_loss‚2.8140632033348085 _step‚736 /loss/FRt1/assignment_loss‚2.826131968787222 0loss/FBt1/assignment_loss‚0.1473637068278549 1loss/FBt2/assignment_loss‚0.18474849588925066 0loss/SRqqt2/detection_loss‚2.151746561129888 1loss/SRqqt3/detection_loss‚0.8702193274039705 % _timestamp‚1.7885353949685004e+09 _runtime‚10778.503767885 0loss/SRqqt1/detection_loss‚2.432548201084137 2loss/SRqqt3/assignment_loss‚0.8801425510714322 /loss/FRt2/detection_loss‚2.5226150252550332 /loss/FRt4/detection_loss‚2.1884415289577173 /loss/FBt3/detection_loss‚0.3980803095975951 /loss/FRt4/assignment_loss‚3.099150666168758 2loss/SRqqt4/assignment_loss‚1.0737465342570995 /loss/FRt1/detection_loss‚2.7883351914810413 1loss/SRqqt4/detection_loss‚2.2120416421314766 trainer/global_step‚8949 0loss/FRt3/assignment_loss‚1.1076972461663759 2loss/SRqqt1/assignment_loss‚1.4014532905817032 1loss/SRqqt2/assignment_loss‚1.513195600360632 1loss/FBt3/assignment_loss‚0.08965080899873357 .loss/FBt2/detection_loss‚1.928332057264116 epoch‚7 0loss/FRt2/assignment_loss‚3.7230122813156674 /loss/FBt4/detection_loss‚1.9187948107719421 %loss/total_loss‚39.87223887653407àÀo8Ê“´&‚(:ˆ éÄëÔØƃá gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.114' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.5285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lè|¶yµ&‚(:î éÄëÔŸ§Óá( disk./.usagePercent‚76.0621842421607 disk.sda1.out‚0 disk.md0.out‚ 1619.06640625 disk.sdb2.in‚ 70.9609375" disk.nvme1n1.out‚ 52924.5546875 proc.cpu.threads‚17 disk.rbd2.out‚ 3.025390625! disk.sda.out‚1662.16455078125# disk./.usageGB‚2720.35213470459 disk.nvme0n1.in‚ 7537.84375" disk.sdb2.out‚1662.16455078125 network.recv‚ 3.531463e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375! proc.memory.rssMB‚ 2977.140625 cpu‚25.22228503543175 disk.sdb1.in‚0 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 8055.95703125 disk.dm-0.out‚ 185409.21875 disk.md0p1.out‚ 1619.06640625$ memory_percent‚91.99535846710205) proc.memory.availableMB‚ 1311.48046875* proc.memory.percent‚18.171024322509766 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.sda2.in‚ 128.6328125! disk.sdb.out‚1662.16455078125 disk.sda.in‚ 128.6328125 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0# disk.nvme0n1.out‚132484.6640625" disk.sda2.out‚1662.16455078125 disk.nvme1n1.in‚ 518.11328125 network.sent‚ 2.376811e+07)cöx¶&‚(:ô øÄëÔÅàöß disk.nvme0n1.in‚ 7537.84375! disk.sda.out‚1664.11279296875 disk.nvme1n1.in‚ 518.11328125 network.sent‚ 2.3881915e+07 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625# disk.nvme1n1.out‚52924.69921875$ disk./.usageGB‚2720.351848602295 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.025390625# disk.nvme0n1.out‚132755.6171875 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1620.96484375 disk.md0p1.out‚ 1620.96484375 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sda2.out‚1664.11279296875# memory_percent‚91.9968843460083 network.recv‚ 3.544258e+06 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1664.11279296875 disk.sda.in‚ 128.6328125 disk.md0.in‚ 199.59375 cpu‚25.222008496097136( disk./.usagePercent‚76.0621762426208 disk.md0p1.in‚ 199.59375) proc.memory.availableMB‚ 1311.23046875* proc.memory.percent‚18.171024322509766! disk.dm-0.out‚185680.31640625 disk.dm-0.in‚ 8055.95703125" disk.sdb2.out‚1664.11279296875 disk.rbd0.out‚ 3.087890625Fsr”·&‚(:‰ øÄëÔÉÂÉà gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·“k”¸&‚(:‰ ‡ÅëÔ‘ŽÎà gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò7c±k¹&‚(:à ‡ÅëÔõ¨²á disk.sdb1.in‚0 disk.sdb.in‚ 70.9609375( proc.memory.availableMB‚ 1311.4765625 network.recv‚ 3.548095e+06% disk./.usageGB‚2720.0744018554688 disk.rbd0.in‚0$ disk.nvme0n1.out‚132794.61328125* proc.memory.percent‚18.171024322509766 disk.md0p1.out‚ 1622.4609375 disk.sda2.in‚ 128.6328125 disk.sda2.out‚ 1665.671875 disk.md0.in‚ 199.59375$ memory_percent‚91.99538230895996 cpu‚25.221740269155557 disk.sda1.in‚0 disk.md0p1.in‚ 199.59375 disk.md0.out‚ 1622.4609375 network.sent‚ 2.3885143e+07 disk.dm-0.out‚185719.6171875 disk.sdb.out‚ 1665.671875 disk.sdb2.out‚ 1665.671875 disk.sdb1.out‚0) disk./.usagePercent‚76.05441871545905 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.out‚ 3.166015625 disk.nvme0n1.in‚ 7537.84375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.041015625 disk.dm-0.in‚ 8055.95703125 disk.sda.in‚ 128.6328125! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17# disk.nvme1n1.out‚52925.00390625 disk.sdb2.in‚ 70.9609375 disk.sda.out‚ 1665.671875TPªº”º&‚(:‰ –ÅëÔÞ¾¾à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GðÒ|»&‚(:ñ –ÅëÔ–£×à$ disk./.usageGB‚2720.010353088379 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.84375 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 disk.sda1.in‚0 disk.dm-0.in‚ 8055.95703125 disk.md0p1.out‚ 1623.8046875! disk.sdb.out‚1667.06787109375" disk.sdb2.out‚1667.06787109375# disk.nvme1n1.out‚52925.19140625( proc.memory.availableMB‚ 1311.4765625 network.recv‚ 3.551904e+06 disk.rbd2.out‚ 3.103515625$ disk.nvme0n1.out‚132796.02734375 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 network.sent‚ 2.3890871e+07" disk.sda2.out‚1667.06787109375 disk.md0.in‚ 199.59375 proc.cpu.threads‚17 disk.md0.out‚ 1623.8046875 disk.dm-0.out‚ 185721.21875! disk.sda.out‚1667.06787109375 disk.rbd0.out‚ 3.166015625 disk.sdb1.in‚0$ memory_percent‚91.99538230895996! proc.memory.rssMB‚ 2977.140625 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375) disk./.usagePercent‚76.05262788512474 disk.nvme1n1.in‚ 518.11328125 disk.sdb1.out‚0 cpu‚25.221460320261514‹ÜF‡4¼&/ _wandbruntime‚10830  _runtime‚10830œÕàÍȽ& & lr-AdamW/pg1‚0.0001362931019098696 & lr-AdamW/pg2‚0.0001362931019098696 trainer/global_step‚8999 % _timestamp‚1.7885354466670942e+09 _runtime‚10830.202617523 _step‚737ûk²o;&Ç trainer/global_step‚8999 % _timestamp‚1.7885354466670942e+09 _runtime‚10830.202617523 _step‚737 & lr-AdamW/pg1‚0.0001362931019098696 & lr-AdamW/pg2‚0.0001362931019098696áÎt4¿&/ _wandbruntime‚10830  _runtime‚10830 x£ý(À&P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768836183981056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758524689208832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _runtime‚10830.203755608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54596161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227972377 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167405676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188275708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227998721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚836866321692160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227998721 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112808892 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311512775168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚738 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10767326965248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733418097897968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚768836183981056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743691982472051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758524671205888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311512775168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758524671205888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10272959559719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54596784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464ma’u (Á&†P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚768837111955456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚758524689208832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚836866345632256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10273867534723 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112808890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416248085 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768836183981056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758524689208832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _runtime‚10830.203755608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54596161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227972377 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167405676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188275708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227998721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚836866321692160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227998721 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112808892 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311512775168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚738 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10767326965248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733418097897968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚768836183981056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743691982472051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758524671205888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311512775168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758524671205888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10272959559719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54596784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768918924527104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167422684 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227997208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚836956571336704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚847724811271680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10768498935296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788535447613574e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188295868 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10313544220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactQáyf5ive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10768263875072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743772117340081 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188295868 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416293076 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188295870 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚743771192325509 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758606308294656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416319448 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228023578 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416319448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167423296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚847725070272000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227997208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733497115492512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10274984808209 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54601729 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10274076832997 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚758606290291712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0·ç¹g'Ä&N PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188295868 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10313544220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10768263875072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743772117340081 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188295868 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416293076 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188295870 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚743771192325509 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758606308294656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416319448 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228023578 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416319448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167423296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚847725070272000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227997208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733497115492512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10274984808209 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54601729 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10274076832997 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚758606290291712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112820955 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768918924527104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167422684 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227997208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚836956571336704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚847724811271680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10768498935296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788535447613574e+09ã,++4Å&/ _wandbruntime‚10831  _runtime‚10831 +bM_Æ&Ù .loss/FRt2/detection_loss‚2.579202127179607 /loss/FBt3/detection_loss‚0.6015706280757841 2loss/SRqqt1/assignment_loss‚1.7409542279584067 2loss/SRqqt2/assignment_loss‚1.5022475825001795 2loss/SRqqt3/assignment_loss‚0.9546052074304434 1loss/FBt2/assignment_loss‚0.17054019802091144 0loss/SRqqt2/detection_loss‚2.231540870666504 1loss/SRqqt4/detection_loss‚1.9959986320563725 _step‚740 /loss/FBt1/detection_loss‚3.3652442267962863 &loss/total_loss‚40.367992383144696 .loss/FRt4/detection_loss‚2.351396871561354 0loss/SRqqt1/detection_loss‚2.040674776690347 /loss/FBt4/detection_loss‚2.1184413616473856 epoch‚7 trainer/global_step‚8999 0loss/FRt1/assignment_loss‚2.7936757295219987 1loss/FBt3/assignment_loss‚0.02480159306012436 0loss/FRt2/assignment_loss‚3.0317227989435196 3loss/FBt1/assignment_loss‚0.0007688580113348767 1loss/SRqqt3/detection_loss‚0.8502372481831999 1loss/SRqqt4/assignment_loss‚1.217610397509166 .loss/FBt2/detection_loss‚2.161420287830489 0loss/FRt4/assignment_loss‚2.9944029222835193 2loss/FBt4/assignment_loss‚0.036821828862614966 .loss/FRt3/detection_loss‚1.258312825113535 $ _timestamp‚1.788535447616103e+09 _runtime‚10831.151374119 0loss/FRt3/assignment_loss‚1.1837080074595165 .loss/FRt1/detection_loss‚3.162093175782098fÕýjdÇ&Þ 0loss/FRt2/assignment_loss‚3.0317227989435196 3loss/FBt1/assignment_loss‚0.0007688580113348767 1loss/SRqqt3/detection_loss‚0.8502372481831999 1loss/SRqqt4/assignment_loss‚1.217610397509166 .loss/FBt2/detection_loss‚2.161420287830489 0loss/FRt4/assignment_loss‚2.9944029222835193 2loss/FBt4/assignment_loss‚0.036821828862614966 .loss/FRt3/detection_loss‚1.258312825113535 $ _timestamp‚1.788535447616103e+09 _runtime‚10831.151374119 0loss/FRt3/assignment_loss‚1.1837080074595165 .loss/FRt1/detection_loss‚3.162093175782098 .loss/FRt2/detection_loss‚2.579202127179607 /loss/FBt3/detection_loss‚0.6015706280757841 2loss/SRqqt1/assignment_loss‚1.7409542279584067 2loss/SRqqt2/assignment_loss‚1.5022475825001795 2loss/SRqqt3/assignment_loss‚0.9546052074304434 1loss/FBt2/assignment_loss‚0.17054019802091144 0loss/SRqqt2/detection_loss‚2.231540870666504 1loss/SRqqt4/detection_loss‚1.9959986320563725 _step‚740 /loss/FBt1/detection_loss‚3.3652442267962863 &loss/total_loss‚40.367992383144696 .loss/FRt4/detection_loss‚2.351396871561354 0loss/SRqqt1/detection_loss‚2.040674776690347 /loss/FBt4/detection_loss‚2.1184413616473856 epoch‚7 trainer/global_step‚8999 0loss/FRt1/assignment_loss‚2.7936757295219987 1loss/FBt3/assignment_loss‚0.02480159306012436äÏ Œ¤”È&‚(:‰ ¥ÅëÔöœºà gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sð]\‚É&‚(:÷ ¥ÅëÔÊê¥á disk.md0p1.out‚ 1625.58203125 disk.dm-0.in‚ 8055.95703125" disk.sdb2.out‚1668.91064453125 disk.sda.in‚ 128.6328125 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.in‚0 disk.md0.out‚ 1625.58203125! disk.sdb.out‚1668.91064453125 disk.dm-0.out‚ 185790.484375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 3.103515625 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 cpu‚25.22117467802208% disk./.usageGB‚2720.0745811462402 disk.md0p1.in‚ 199.59375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 70.9609375$ memory_percent‚91.99693202972412 disk.sdb1.in‚0$ disk.nvme0n1.out‚132864.98046875" disk.sda2.out‚1668.91064453125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 network.recv‚ 3.556402e+06) disk./.usagePercent‚76.05442372850406 disk.rbd0.out‚ 3.25341796875 disk.md0.in‚ 199.59375! disk.sda.out‚1668.91064453125) proc.memory.availableMB‚ 1311.22265625# disk.nvme1n1.out‚52925.50390625 network.sent‚ 2.3909859e+07H.)ãƒÊ&‚(:ø ´ÅëÔ›¹£à disk.md0p1.in‚ 199.59375 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 cpu‚25.220851870784482 network.recv‚ 3.561992e+06$ disk./.usageGB‚2720.261894226074 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7537.84375 disk.md0.out‚ 1627.0703125! disk.dm-0.out‚185860.64453125" disk.sdb2.out‚1670.44873046875! disk.sda.out‚1670.44873046875" disk.sda2.out‚1670.44873046875# disk.nvme1n1.out‚52925.66796875) proc.memory.availableMB‚ 1310.98046875 network.sent‚ 2.4005202e+07) disk./.usagePercent‚76.05966108061284 disk.rbd0.in‚0 disk.md0.in‚ 199.59375 disk.dm-0.in‚ 8055.95703125! disk.sdb.out‚1670.44873046875$ memory_percent‚91.99841022491455 disk.rbd0.out‚ 3.25341796875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.19091796875 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.171024322509766 disk.md0p1.out‚ 1627.0703125 disk.sda1.in‚0# disk.nvme0n1.out‚132934.9765625 disk.nvme1n1.in‚ 518.11328125 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17[g¥ÄË&‚(:’ ´ÅëÔʼÛà gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š“ô“’Ì&‚(:‡ ÃÅëÔîîáÞ gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’±(”~Í&‚(:ó ÃÅëÔãó×ß disk.dm-0.in‚ 8055.95703125! disk.sda.out‚1671.86962890625# disk.nvme1n1.out‚52926.10546875 disk.rbd0.in‚0 disk.md0p1.out‚ 1628.42578125* proc.memory.percent‚18.171024322509766 disk.rbd0.out‚ 3.31640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 128.6328125" disk.sdb2.out‚1671.86962890625 disk.sda.in‚ 128.6328125) disk./.usagePercent‚76.05965969402591% disk./.usageGB‚2720.2618446350098$ disk.nvme0n1.out‚133049.08203125 disk.md0.in‚ 199.59375 disk.dm-0.out‚ 185975.1875 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0$ memory_percent‚91.99709892272949! proc.memory.rssMB‚ 2977.140625 cpu‚25.220550767619283" disk.sda2.out‚1671.86962890625! disk.sdb.out‚1671.86962890625 disk.nvme1n1.in‚ 518.11328125 network.recv‚ 3.56574e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.19091796875 disk.md0p1.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 disk.md0.out‚ 1628.42578125( proc.memory.availableMB‚ 1311.1953125 proc.cpu.threads‚17 network.sent‚ 2.4010647e+07ŽÙ”*4Î&/ _wandbruntime‚10881  _runtime‚10881Áá:wÈÏ& ' lr-AdamW/pg1‚0.00013220443001153782 ' lr-AdamW/pg2‚0.00013220443001153782 trainer/global_step‚9049 $ _timestamp‚1.788535497478217e+09 _runtime‚10881.01372704 _step‚741G–C¶ÍÐ&Ç ' lr-AdamW/pg1‚0.00013220443001153782 ' lr-AdamW/pg2‚0.00013220443001153782 trainer/global_step‚9049 $ _timestamp‚1.788535497478217e+09 _runtime‚10881.01372704 _step‚741å’‹ Ã4Ñ&/ _wandbruntime‚10881  _runtime‚10881yž n "Ò&P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229217279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772973225697280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113412142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747697814439995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418500987 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772973225697280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54886014 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54885374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189283708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10825708215296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚762605643500032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚841377606914560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229243623 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚9049 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113412140 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚762605625497088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762605625497088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10368510171648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189283708 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747698739454359 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚737368977625168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229243623 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10367600200192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10329744789831 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10881.014699094 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚762605643500032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772974153671680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189283710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 ZFDeviceStatsMonitor.on_train_batch_start/inU…?Àûactive_split_bytes.all.freed‚852203071667712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168297514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚737368994664528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229217279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885354974791074e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10881.014699094 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚762605643500032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772974153671680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189283710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚852203071667712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168297514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚737368994664528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229217279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885354974791074e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772973225697280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113412142 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747697814439995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418500987 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772973225697280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54886014 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54885374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189283708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10825708215296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚762605643500032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚841377606914560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229243623 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚9049 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113412140 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚762605625497088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762605625497088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10368510171648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189283708 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747698739454359 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚737368977625168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570æϺ.—4Ô&/ _wandbruntime‚10881  _runtime‚10881Á©¹é'Õ&‹N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189303870 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189303868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852294737758208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚762687262585856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10368721660416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229268480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189303870 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚743 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10368721660416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚773056894231040 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168315353 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841467856559104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚773055966243328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852294499729408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229268480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚737447995219712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚773056894231040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762687244582912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54891148 _runtime‚10881.994130725 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762687262585856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚737448012259072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229242110 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189303868 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418572350 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168315970 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10826881199104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚9049 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229242110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚841467832619008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10330862063317 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10369631645184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762687244582912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10826667110400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329954088105 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418545978 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0@'Ö&N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189303870 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚743 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10368721660416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚773056894231040 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168315353 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841467856559104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚773055966243328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852294499729408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229268480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚737447995219712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚773056894231040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762687244582912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54891148 _runtime‚10881.994130725 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762687262585856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚737448012259072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229242110 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189303868 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418572350 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168315970 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10826881199104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚9049 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229242110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚841467832619008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10330862063317 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10369631645184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762687244582912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10826667110400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329954088105 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418545978 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189303870 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189303868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852294737758208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚762687262585856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10368721660416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229268480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ç;¨4×&/ _wandbruntime‚10881  _runtime‚10881AÅ;M9Ø&Ö 0loss/FRt3/assignment_loss‚0.8476119270691505 -loss/FRt2/detection_loss‚2.73984166921517 _runtime‚10881.99520518 2loss/SRqqt3/assignment_loss‚0.8758986416537113 1loss/SRqqt3/detection_loss‚0.7402634030996679 /loss/SRqqt4/detection_loss‚2.07159718997935 .loss/FBt1/detection_loss‚2.134918663650751 /loss/FBt2/detection_loss‚2.4965383807818093 trainer/global_step‚9049 _step‚744 2loss/SRqqt2/assignment_loss‚1.4655039387530293 0loss/SRqqt1/detection_loss‚2.232111672560374 0loss/FRt1/assignment_loss‚3.2074092483520507 1loss/FBt2/assignment_loss‚0.18118583635886173 1loss/FBt3/assignment_loss‚0.07876640687385683 /loss/FRt1/detection_loss‚2.9180302810668945 0loss/FBt3/detection_loss‚0.41198116291314363 &loss/total_loss‚40.626177013413184 % _tÉ»ú-#imestamp‚1.7885354984599302e+09 0loss/FBt4/assignment_loss‚0.1471540601444249 /loss/FRt4/detection_loss‚2.4098882090462292 /loss/FRt4/assignment_loss‚2.911759976459586 1loss/SRqqt2/detection_loss‚2.5318252279840667 1loss/FBt1/assignment_loss‚0.17873712056805857 .loss/FRt3/detection_loss‚1.293561574128958 /loss/FBt4/detection_loss‚1.8418880473164951 epoch‚7 0loss/FRt2/assignment_loss‚3.8650143329913798 2loss/SRqqt1/assignment_loss‚1.5517322902878126 2loss/SRqqt4/assignment_loss‚1.4929577521583501_ÖÞËaÙ&Û 0loss/FRt1/assignment_loss‚3.2074092483520507 1loss/FBt2/assignment_loss‚0.18118583635886173 1loss/FBt3/assignment_loss‚0.07876640687385683 /loss/FRt1/detection_loss‚2.9180302810668945 0loss/FBt3/detection_loss‚0.41198116291314363 &loss/total_loss‚40.626177013413184 % _timestamp‚1.7885354984599302e+09 0loss/FBt4/assignment_loss‚0.1471540601444249 /loss/FRt4/detection_loss‚2.4098882090462292 /loss/FRt4/assignment_loss‚2.911759976459586 1loss/SRqqt2/detection_loss‚2.5318252279840667 1loss/FBt1/assignment_loss‚0.17873712056805857 .loss/FRt3/detection_loss‚1.293561574128958 /loss/FBt4/detection_loss‚1.8418880473164951 epoch‚7 0loss/FRt2/assignment_loss‚3.8650143329913798 2loss/SRqqt1/assignment_loss‚1.5517322902878126 2loss/SRqqt4/assignment_loss‚1.4929577521583501 0loss/FRt3/assignment_loss‚0.8476119270691505 -loss/FRt2/detection_loss‚2.73984166921517 _runtime‚10881.99520518 2loss/SRqqt3/assignment_loss‚0.8758986416537113 1loss/SRqqt3/detection_loss‚0.7402634030996679 /loss/SRqqt4/detection_loss‚2.07159718997935 .loss/FBt1/detection_loss‚2.134918663650751 /loss/FBt2/detection_loss‚2.4965383807818093 trainer/global_step‚9049 _step‚744 2loss/SRqqt2/assignment_loss‚1.4655039387530293 0loss/SRqqt1/detection_loss‚2.232111672560374èÂWIÔÚ&‚(:‰ ÒÅëÔãè©á gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùavÛ&‚(:ë ÒÅëÔŒ¶èá disk.sda.out‚1673.3388671875 disk.nvme1n1.in‚ 518.11328125 disk.rbd2.out‚ 3.25390625 disk.nvme0n1.in‚ 7537.84375$ memory_percent‚91.99893474578857) proc.memory.availableMB‚ 1310.89453125 disk.md0p1.out‚ 1629.83984375! disk.sdb2.out‚1673.3388671875 network.recv‚ 3.570158e+06$ disk./.usageGB‚2720.398105621338 disk.md0.out‚ 1629.83984375 disk.sdb.in‚ 70.9609375* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 128.6328125" disk.nvme0n1.out‚ 133256.390625 disk.md0.in‚ 199.59375 cpu‚25.22023734827921 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 2.4016916e+07 disk.dm-0.out‚186182.7421875 disk.sdb.out‚1673.3388671875( disk./.usagePercent‚76.0634696082333" disk.nvme1n1.out‚ 52926.3515625 disk.sda2.in‚ 128.6328125 disk.sdb2.in‚ 70.9609375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.31640625! disk.sda2.out‚1673.3388671875! proc.memory.rssMB‚ 2977.140625µxóð’Ü&‚(:‡ áÅëÔ°Ïãà gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®”ŒyÝ&‚(:î áÅëÔû £á# memory_percent‚91.9989824295044 cpu‚25.219937678661573 network.recv‚ 3.583904e+06 disk.sdb.out‚1675.4541015625 disk.sda.out‚1675.4541015625 disk.nvme1n1.in‚ 518.11328125 disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375) disk./.usagePercent‚76.06175760003222 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.sdb1.in‚0 disk.rbd2.out‚ 3.25390625% disk./.usageGB‚2720.3368759155273 disk.rbd2.in‚0 disk.sda2.in‚ 128.6328125! disk.sda2.out‚1675.4541015625) proc.memory.availableMB‚ 1310.88671875 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 network.sent‚ 2.413067e+07 disk.rbd0.in‚0! proc.memory.rssMB‚ 2977.140625 disk.md0.in‚ 199.59375 disk.md0p1.out‚ 1631.8984375 disk.sdb1.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚133463.79296875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17! disk.sdb2.out‚1675.4541015625# disk.nvme1n1.out‚52926.55078125 disk.rbd0.out‚ 3.37939453125 disk.md0.out‚ 1631.8984375 disk.dm-0.out‚ 186390.34375 Ä}B“Þ&‚(:ˆ ðÅëÔ‡ðîÞ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zˆK$ß&‚(:ô ðÅëÔ—Ï·à! disk.sda.out‚1676.75439453125 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.out‚ 1633.140625# memory_percent‚91.9959306716919! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.4133696e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚186433.48828125 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 cpu‚25.2196631930099$ disk.nvme0n1.out‚133506.80859375" disk.sda2.out‚1676.75439453125" disk.sdb2.out‚1676.75439453125% disk./.usageGB‚2720.1874656677246 disk.sda1.in‚0 disk.rbd2.out‚ 3.31689453125 disk.sda2.in‚ 128.6328125* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 network.recv‚ 3.587302e+06 disk.sda1.out‚0 disk.md0.out‚ 1633.140625! disk.sdb.out‚1676.75439453125 disk.md0p1.in‚ 199.59375) proc.memory.availableMB‚ 1311.38671875 disk.rbd0.in‚0 disk.md0.in‚ 199.59375) disk./.usagePercent‚76.05758002697061 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 52926.6796875 disk.rbd0.out‚ 3.37939453125 disk.nvme0n1.in‚ 7537.84375µ””—”à&‚(:‰ ÿÅëÔ˜¿éÞ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.509' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.37725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÈWò{á&‚(:ð ÿÅëÔ£™ßß disk.dm-0.in‚ 8055.95703125! disk.dm-0.out‚186594.33984375 disk.rbd2.out‚ 3.31689453125 disk.nvme0n1.in‚ 7537.84375 disk.sda2.out‚1678.369140625 disk.md0.out‚ 1634.68359375) proc.memory.availableMB‚ 1311.12890625 cpu‚25.21936579300982 disk.sdb.out‚1678.369140625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.md0p1.out‚ 1634.68359375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda.out‚1678.369140625$ memory_percent‚91.99750423431396 network.sent‚ 2.4139143e+07 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚133603.3515625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766% disk./.usageGB‚2720.4001121520996 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme1n1.out‚52990.98828125( disk./.usagePercent‚76.0635257116732 disk.sdb1.out‚0 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.out‚ 3.37939453125 disk.md0.in‚ 199.59375 disk.md0p1.in‚ 199.59375 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚1678.369140625 network.recv‚ 3.59136e+06 disk.sda2.in‚ 128.6328125ðéž4â&/ _wandbruntime‚10937  _runtime‚10937Ÿ‡¨ÖÉã&à ' lr-AdamW/pg1‚0.00012816445986500865 ' lr-AdamW/pg2‚0.00012816445986500865 trainer/global_step‚9099 $ _timestamp‚1.788535553486455e+09 _runtime‚10937.021982231 _step‚745]ä¶Îä&È trainer/global_step‚9099 $ _timestamp‚1.788535553486455e+09 _runtime‚10937.021982231 _step‚745 ' lr-AdamW/pg1‚0.00012816445986500865 ' lr-AdamW/pg2‚0.00012816445986500865éÊ9,…4å&/ _wandbruntime‚10937  _runtime‚10937-Â_ (æ&P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55175530 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190291710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190291710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845888916077056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751704571422303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10385622044939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230462181 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚741319857352368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777110267413504 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777110267413504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420780235 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10423687625216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190291708 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845888892136960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751705496436667 % _timestamp‚1.7885355534872231e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766686579788288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230462181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169190920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚9099 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55176163 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10884369802240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114015390 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230488525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚741319874391728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚856773285879296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10384714069935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚777111195387904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230488525 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169191555 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚777111195387904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _runtime‚10937.022809865 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766686597791232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856773031059968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766686579788288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114015392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640ÏGW (ç&†P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766686579788288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114015392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55175530 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190291710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190291710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845888916077056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751704571422303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10385622044939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230462181 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚741319857352368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777110267413504 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777110267413504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420780235 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10423687625216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190291708 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845888892136960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751705496436667 % _timestamp‚1.7885355534872231e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766686579788288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230462181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169190920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚9099 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55176163 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10884369802240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114015390 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230488525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚741319874391728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚856773285879296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10384714069935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚777111195387904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230488525 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169191555 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚777111195387904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _runtime‚10937.022809865 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766686597791232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856773031059968êõãê4è&/ _wandbruntime‚10938  _runtime‚10938Ý«Y#‚ é&‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420825252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190311870 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10425719070208 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114027457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚777193007959552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10424809085440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751784706290125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10424809085440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169208791 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420825252 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766768216877056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751785631304697 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚741398874946912 _step‚747 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55181336 _runtime‚10938.036656997 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55181976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚777193935947264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚777193935947264 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230487012 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10386739318425 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190311870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚741398891986272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230513382 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚856864426834944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10885308993536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0½Yò'ê&N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚845979141781504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10424809085440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169208791 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420825252 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766768216877056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751785631304697 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚741398874946912 _step‚747 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55181336 _runtime‚10938.036656997 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55181976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚777193935947264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚777193935947264 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230487012 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10386739318425 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190311870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚741398891986272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230513382 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚856864426834944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10885308993536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420825252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190311870 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10425719070208 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114027457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚777193007959552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10424809085440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751784706290125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781247309129728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10479775050240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚781247309129728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚770767534079488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚781248237104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423006791 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231707083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10942586824192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231707083 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10480685021696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚10991.875289269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10480685021696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚861342764183552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781248237104128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191299710 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781247309129728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10479775050240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚781247309129728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚770767534079488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚781248237104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423006791 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231707083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10942586824192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231707083 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10480685021696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚10991.875289269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10480685021696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚861342764183552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781248237104128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191299710 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚770849171168256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231731914 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861434425193984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚770849153165312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55470707 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚10992.859351781 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191319870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10441708598321 $ _timestamp‚1.788535609323388e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191319870 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚781330049675776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10943998190080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚755792388287005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚9149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10442616573533 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170101412 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10481806495232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770849153165312 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114630705 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚781330977663488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 423051782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850490403063808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚770849171168256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10943779907072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745349771713472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231731914 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10481806495232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423078154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231758284 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114630707 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10480896510464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚850490427003904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640TG^ã'ü&N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10480896510464 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 423078154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114630707 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10480896510464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚850490427003904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55471313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚745349754674112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚781330977663488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191319868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 423051782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚861434182970880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚770849171168256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231731914 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861434425193984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚770849153165312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55470707 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚10992.859351781 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191319870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10441708598321 $ _timestamp‚1.788535609323388e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191319870 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚781330049675776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10943998190080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚755792388287005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚9149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10442616573533 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170101412 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10481806495232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770849153165312 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114630705 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚781330977663488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 423051782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850490403063808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚770849171168256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10943779907072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745349771713472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231731914 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10481806495232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423078154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231758284 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785385278820352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774848506373632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11001304196096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785384350845952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55755085 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536772446720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854911462581760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232951985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232951985 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170976365 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788535665815888e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774848488370688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115221892 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚749221633846128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10497376555155 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536772446720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192307710 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785385278820352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774848506373632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11001304196096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785384350845952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55755085 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536772446720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854911462581760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232951985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232951985 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170976365 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788535665815888e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774848488370688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115221892 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚749221633846128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10497376555155 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536772446720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231òÊËÿa4Ž'/ _wandbruntime‚11050  _runtime‚11050 Àw™> '‹N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785468019379712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192327870 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192327868 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785468019379712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10498493828641 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232976816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚9199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55760902 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚759799145269313 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536983935488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115233957 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10497585853429 B9DeviÀø¿ ÓceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425331056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866004411243520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774930107456512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425331056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55760295 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170994250 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425304684 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537893920256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855001688286208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11002466054144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785467091392000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192327868 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774930125459456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425304684 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚866004154340352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚759799145269313 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536983935488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115233957 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10497585853429 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425331056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866004411243520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774930107456512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425331056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55760295 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170994250 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425304684 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537893920256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855001688286208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11002466054144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785467091392000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192327868 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774930125459456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425304684 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚866004154340352óÖ™bˆ4‘'/ _wandbruntime‚11050  _runtime‚11050šh+c’'Ý 0loss/FRt2/assignment_loss‚3.0359207498175755 /loss/FRt4/assignment_loss‚3.385487055524867 2loss/SRqqt3/assignment_loss‚0.9789504888942655 _runtime‚11050.504100462 2loss/SRqqt2/assignment_loss‚1.3517442277736134 1loss/SRqqt1/detection_loss‚2.4022874811599992 1loss/SRqqt3/detection_loss‚0.9091451043567184 /loss/FBt1/detection_loss‚2.8909996492522105 2loss/SRqqt1/assignment_loss‚1.4028308091942092 1loss/FBt2/assignment_loss‚0.20191133961974023 /loss/FRt3/detection_loss‚1.4870403094537323 &loss/total_loss‚41.363682469336325 2loss/FBt1/assignment_loss‚0.013470201760778113 0loss/FBt3/assignment_loss‚0.0323844330527534 1loss/FBt4/assignment_loss‚0.17618922559475436 .loss/FRt1/detection_loss‚3.153705749511719 1loss/SRqqt4/detection_loss‚2.1043791994452477 /loss/FRt4/detection_loss‚2.3150842772360813 /loss/FBt4/detection_loss‚2.4026438551289693 epoch‚7 _step‚756 0loss/FRt3/assignment_loss‚1.4682516492903233 1loss/SRqqt2/detection_loss‚1.8475394172800912 0loss/FBt3/detection_loss‚0.42387847491374164 trainer/global_step‚9199 2loss/SRqqt4/assignment_loss‚1.1225887623149902 /loss/FRt2/detection_loss‚2.4179337010562967 $ _timestamp‚1.788535666968748e+09 0loss/FRt1/assignment_loss‚3.2439421129226687 /loss/FBt2/detection_loss‚2.5953741947809856^s#nh“'â 1loss/FBt4/assignment_loss‚0.17618922559475436 .loss/FRt1/detection_loss‚3.153705749511719 1loss/SRqqt4/detection_loss‚2.1043791994452477 /loss/FRt4/detection_loss‚2.3150842772360813 /loss/FBt4/detection_loss‚2.4026438551289693 epoch‚7 _step‚756 0loss/FRt3/assignment_loss‚1.4682516492903233 1loss/SRqqt2/detection_loss‚1.8475394172800912 0loss/FBt3/detection_loss‚0.42387847491374164 trainer/global_step‚9199 2loss/SRqqt4/assignment_loss‚1.1225887623149902 /loss/FRt2/detection_loss‚2.4179337010562967 $ _timestamp‚1.788535666968748e+09 0loss/FRt1/assignment_loss‚3.2439421129226687 /loss/FBt2/detection_loss‚2.5953741947809856 0loss/FRt2/assignment_loss‚3.0359207498175755 /loss/FRt4/assignment_loss‚3.385487055524867 2loss/SRqqt3/assignment_loss‚0.9789504888942655 _runtime‚11050.504100462 2loss/SRqqt2/assignment_loss‚1.3517442277736134 1loss/SRqqt1/detection_loss‚2.4022874811599992 1loss/SRqqt3/detection_loss‚0.9091451043567184 /loss/FBt1/detection_loss‚2.8909996492522105 2loss/SRqqt1/assignment_loss‚1.4028308091942092 1loss/FBt2/assignment_loss‚0.20191133961974023 /loss/FRt3/detection_loss‚1.4870403094537323 &loss/total_loss‚41.363682469336325 2loss/FBt1/assignment_loss‚0.013470201760778113 0loss/FBt3/assignment_loss‚0.0323844330527534ôú¶¾””'‚(:‰ ÷ÆëÔ’ðêá gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Ó}•'‚(:ò ÷ÆëÔïÔùâ! disk.sdb.out‚1691.51611328125 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 518.11328125 disk.nvme1n1.out‚ 53452.40625 disk.sda1.in‚0! disk.dm-0.out‚187973.80078125 disk.rbd0.out‚ 3.37939453125 disk.rbd2.out‚ 3.31689453125$ disk.nvme0n1.out‚134521.39453125! disk.sda.out‚1691.51611328125$ disk./.usageGB‚2721.277240753174 disk.md0.in‚ 199.59375 disk.md0.out‚ 1647.359375" disk.sdb2.out‚1691.51611328125 disk.md0p1.in‚ 199.59375) disk./.usagePercent‚76.08805059446607 disk.sdb1.out‚0 disk.sda1.out‚0" disk.sda2.out‚1691.51611328125$ memory_percent‚91.99824333190918 network.sent‚ 2.439769e+07 disk.sda.in‚ 128.6328125 disk.rbd2.in‚0 network.recv‚ 3.645065e+06 disk.md0p1.out‚ 1647.359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7537.84375( proc.memory.availableMB‚ 1311.0078125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 70.9609375 disk.sdb1.in‚0 disk.sda2.in‚ 128.6328125 cpu‚25.217290441084348 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125¼÷Ø-’–'‚(:‡ †ÇëÔ¦‹»ß gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û<_Lu—'‚(:ê †ÇëÔùÿ‰á disk.dm-0.in‚ 8055.95703125 disk.dm-0.out‚188181.6953125 disk.rbd0.in‚0$ memory_percent‚92.00000762939453 proc.cpu.threads‚17 disk.sdb.out‚1693.029296875 disk.nvme1n1.in‚ 518.11328125 disk.rbd2.out‚ 3.31689453125! proc.memory.rssMB‚ 2977.140625 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 134728.84375& proc.memory.availableMB‚ 1310.71875 network.recv‚ 3.651936e+06 disk.sda2.out‚1693.029296875 network.sent‚ 2.4507819e+07) disk./.usagePercent‚76.09156228582566 disk.md0.in‚ 199.59375 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1648.8203125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.84375 cpu‚25.217050641891163 disk.md0.out‚ 1648.8203125 disk.sdb.in‚ 70.9609375 disk.sda.out‚1693.029296875" disk.nvme1n1.out‚ 53452.8515625 disk.md0p1.in‚ 199.59375 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 128.6328125% disk./.usageGB‚2721.4028358459473 disk.sda2.in‚ 128.6328125 disk.sdb2.out‚1693.029296875 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0+m’˜'‚(:‡ •ÇëԸēâ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¿£™'‚(:ö •ÇëÔ¯·šä cpu‚25.21681055371629 disk.rbd2.out‚ 3.31689453125) disk./.usagePercent‚76.09154575344317 disk.rbd0.in‚0* proc.memory.percent‚18.171024322509766" disk.sda2.out‚1694.54541015625 disk.dm-0.in‚ 8055.95703125$ memory_percent‚91.99841022491455 disk.sdb.in‚ 70.9609375 disk.sda1.out‚0 network.sent‚ 2.4512021e+07 network.recv‚ 3.656758e+06 disk.md0.in‚ 199.59375 disk.rbd2.in‚0$ disk./.usageGB‚2721.402244567871! disk.dm-0.out‚188475.03515625! disk.sdb.out‚1694.54541015625 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1650.2734375 disk.sdb1.out‚0 disk.nvme1n1.in‚ 518.11328125 disk.nvme0n1.in‚ 7537.84375! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17" disk.sdb2.out‚1694.54541015625 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1650.2734375 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125" disk.nvme0n1.out‚ 135022.015625) proc.memory.availableMB‚ 1310.98046875! disk.sda.out‚1694.54541015625# disk.nvme1n1.out‚53453.01953125 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0 disk.sda1.in‚0ØÑmœ’š'‚(:‡ ¤ÇëÔŸòŸà gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿª7y›'‚(:î ¤ÇëÔóøÄá! disk.sda2.out‚1695.9443359375 disk.sdb.in‚ 70.9609375" disk.nvme1n1.out‚ 53453.2578125! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.674545e+06 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1651.6171875( proc.memory.availableMB‚ 1309.3515625 network.sent‚ 2.4524879e+07 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚188475.6015625 disk.sda.in‚ 128.6328125 disk.sda.out‚1695.9443359375) disk./.usagePercent‚76.09154735335115 disk.nvme0n1.in‚ 7537.84375 disk.sdb2.in‚ 70.9609375 disk.md0p1.out‚ 1651.6171875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚ 3.31689453125! disk.sdb2.out‚1695.9443359375 disk.sda1.out‚0! disk.nvme0n1.out‚ 135022.34375 disk.sda2.in‚ 128.6328125 disk.dm-0.in‚ 8055.95703125 disk.nvme1n1.in‚ 518.11328125$ memory_percent‚92.00835227966309# disk./.usageGB‚2721.40230178833 disk.md0.in‚ 199.59375 disk.sdb.out‚1695.9443359375 disk.md0p1.in‚ 199.59375* proc.memory.percent‚18.171024322509766 cpu‚25.216485556972106t)a¼4œ'/ _wandbruntime‚11108  _runtime‚11108FܪEÊ'Ä _step‚757 ' lr-AdamW/pg1‚0.00011634512737602733 ' lr-AdamW/pg2‚0.00011634512737602733 trainer/global_step‚9249 % _timestamp‚1.7885357254101846e+09 _runtime‚11108.946056996«K –Ïž'É % _timestamp‚1.7885357254101846e+09 _runtime‚11108.946056996 _step‚757 ' lr-AdamW/pg1‚0.00011634512737602733 ' lr-AdamW/pg2‚0.00011634512737602733 trainer/global_step‚9249õ€;S94Ÿ'/ _wandbruntime‚11108  _runtime‚11108âëTc 'P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789521392562176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56043411 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚870482582191616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10592859871744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚763724842369227 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚9249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171868551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11059834387456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 193315710 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789522320536576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778929460664832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10592859871744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778929442661888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234223231 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11108.947455503 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753172496533968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚789522320536576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427538941 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11060065266688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193315708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171869173 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115825142 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10591949900288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859422771744256ƒÇô (¡'†P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789522320536576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778929460664832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10592859871744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778929442661888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234223231 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11108.947455503 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753172496533968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚789522320536576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427538941 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11060065266688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193315708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171869173 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115825142 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10591949900288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859422771744256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789521392562176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56043411 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚870482582191616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10592859871744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚763724842369227 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚9249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171868551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11059834387456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 193315710ö8÷\4¢'/ _wandbruntime‚11109  _runtime‚11109up74'£'ŠN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870574238171136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚789604133108224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193335868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427557586 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234221718 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427583958 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193335870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚753251514128512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779011079750656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171886446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11061240722432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10593071360512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚779011061747712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788535726458725e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193335868 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚789605061095936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234248088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10593071360512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10554371083749 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171887055 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859512997448704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763805902251621 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115837207 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427557586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753251531167872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784¾N•Ké¤'N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870574238171136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚789604133108224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193335868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427557586 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234221718 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427583958 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193335870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚753251514128512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779011079750656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171886446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11061240722432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10593071360512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚779011061747712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788535726458725e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193335868 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚789605061095936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234248088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10593071360512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10554371083749 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171887055 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859512997448704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763805902251621 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115837207 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427557586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753251531167872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784÷?èÜG4¥'/ _wandbruntime‚11109  _runtime‚11109¥ò^¦'Ø 0loss/FRt1/assignment_loss‚2.6570134632515185 /loss/FRt1/detection_loss‚2.6185129736409043 /loss/FBt1/detection_loss‚1.9321972865324755 .loss/FBt2/detection_loss‚2.811208561062813 /loss/FRt3/assignment_loss‚0.972748715877533 1loss/FBt4/assignment_loss‚0.27825813409827577 /loss/FRt3/detection_loss‚1.4393684089680512 .loss/FRt4/detection_loss‚2.369585739726082 0loss/SRqqt4/detection_loss‚2.117910554011663 0loss/FRt4/assignment_loss‚2.8863479976958417 2loss/SRqqt3/assignment_loss‚0.8619805020328689 2loss/SRqqt4/assignment_loss‚1.5188736342119449 trainer/global_step‚9249 epoch‚7 % _timestamp‚1.7885357264611187e+09 2loss/SRqqt1/assignment_loss‚1.3472164548360384 1loss/SRqqt2/assignment_loss‚1.340243117474928 1loss/FBt1/assignment_loss‚0.09953701392469856 2loss/FBt3/assignment_loss‚0.041304998977960056 0loss/SRqqt2/detection_loss‚2.123802239575037 /loss/FBt3/detection_loss‚0.5622873547309566 1loss/FBt2/assignment_loss‚0.25463244877391844 1loss/SRqqt3/detection_loss‚0.7812774935155176 .loss/FBt4/detection_loss‚2.511738823010371 _runtime‚11109.996393154 /loss/FRt2/detection_loss‚2.6549421003997202 %loss/total_loss‚39.95224670970586 _step‚760 0loss/FRt2/assignment_loss‚3.1479705674307685 1loss/SRqqt1/detection_loss‚2.6232881259459715ìÜVc§'Ý epoch‚7 % _timestamp‚1.7885357264611187e+09 2loss/SRqqt1/assignment_loss‚1.3472164548360384 1loss/SRqqt2/assignment_loss‚1.340243117474928 1loss/FBt1/assignment_loss‚0.09953701392469856 2loss/FBt3/assignment_loss‚0.041304998977960056 0loss/SRqqt2/detection_loss‚2.123802239575037 /loss/FBt3/detection_loss‚0.5622873547309566 1loss/FBt2/assignment_loss‚0.25463244877391844 1loss/SRqqt3/detection_loss‚0.7812774935155176 .loss/FBt4/detection_loss‚2.511738823010371 _runtime‚11109.996393154 /loss/FRt2/detection_loss‚2.6549421003997202 %loss/total_loss‚39.95224670970586 _step‚760 0loss/FRt2/assignment_loss‚3.1479705674307685 1loss/SRqqt1/detection_loss‚2.6232881259459715 0loss/FRt1/assignment_loss‚2.6570134632515185 /loss/FRt1/detection_loss‚2.6185129736409043 /loss/FBt1/detection_loss‚1.9321972865324755 .loss/FBt2/detection_loss‚2.811208561062813 /loss/FRt3/assignment_loss‚0.972748715877533 1loss/FBt4/assignment_loss‚0.27825813409827577 /loss/FRt3/detection_loss‚1.4393684089680512 .loss/FRt4/detection_loss‚2.369585739726082 0loss/SRqqt4/detection_loss‚2.117910554011663 0loss/FRt4/assignment_loss‚2.8863479976958417 2loss/SRqqt3/assignment_loss‚0.8619805020328689 2loss/SRqqt4/assignment_loss‚1.5188736342119449 trainer/global_step‚9249øø@Í”¨'‚(:‰ ³ÇëÔÖÛ¿à gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø_- }©'‚(:ò ³ÇëÔºèÇá disk.rbd0.in‚0 disk.sdb.in‚ 70.9609375! disk.sda.out‚1697.83935546875 proc.cpu.threads‚17 disk.md0.in‚ 199.59375" disk.sdb2.out‚1697.83935546875 disk.md0p1.out‚ 1653.4453125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.dm-0.in‚ 8055.95703125 disk.sda.in‚ 128.6328125 disk.sda1.in‚0 disk.rbd2.out‚ 3.31689453125" disk.sda2.out‚1697.83935546875 network.recv‚ 3.678262e+06% disk./.usageGB‚2721.1252250671387 disk.sda1.out‚0$ disk.nvme0n1.out‚135429.72265625! disk.dm-0.out‚188883.25390625! disk.sdb.out‚1697.83935546875& proc.memory.availableMB‚ 1310.65625 disk.nvme0n1.in‚ 7537.84375) disk./.usagePercent‚76.08380017226104 cpu‚25.21626128976798 disk.sdb1.out‚0 disk.md0p1.in‚ 199.59375# memory_percent‚92.0003890991211 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 518.11328125 network.sent‚ 2.4530157e+07 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1653.4453125 disk.nvme1n1.out‚ 53453.53125C2|X”ª'‚(:‰ ÂÇëÔàÒŸà gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼«{y«'‚(:î ÂÇëÔ¢¡Øà disk.md0.in‚ 199.59375 disk.dm-0.in‚ 8055.95703125 disk.sdb.out‚1699.4501953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 3.37939453125! disk.sda2.out‚1699.4501953125 disk.rbd0.out‚ 3.37939453125) disk./.usagePercent‚76.05331317904361 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sda.out‚1699.4501953125 disk.nvme1n1.in‚ 518.11328125' proc.memory.availableMB‚ 1309.671875 network.sent‚ 2.4643083e+07 disk.rbd2.in‚0! disk.sdb2.out‚1699.4501953125 disk.md0p1.out‚ 1655.0078125 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 network.recv‚ 3.691066e+06$ memory_percent‚92.00639724731445 cpu‚25.215996927678436 disk.nvme0n1.in‚ 7537.84375$ disk.nvme0n1.out‚135556.23046875 disk.md0.out‚ 1655.0078125 disk.nvme1n1.out‚53453.75 disk.rbd0.in‚0 disk.sda1.in‚0! disk.dm-0.out‚189009.98046875 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375% disk./.usageGB‚2720.0348625183105* proc.memory.percent‚18.171024322509766*¶Ús¬'‚(:è ÑÇëԚņà" disk.nvme0n1.out‚ 135623.046875) disk./.usagePercent‚76.05606427414862 disk.sda2.out‚1700.755859375 disk.sdb2.in‚ 70.9609375 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 70.9609375 disk.md0p1.in‚ 199.59375! disk.nvme1n1.out‚ 53453.921875 network.sent‚ 2.4653268e+07 disk.rbd2.in‚0 disk.sdb.out‚1700.755859375$ memory_percent‚91.99903011322021! proc.memory.rssMB‚ 2977.140625 disk.sdb1.in‚0 disk.rbd2.out‚ 3.37939453125 disk.md0.out‚ 1656.265625 network.recv‚ 3.703129e+06 disk.rbd0.out‚ 3.44189453125 disk.sdb1.out‚0 disk.md0.in‚ 199.59375 disk.dm-0.out‚ 189076.96875 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.in‚0 disk.sda.in‚ 128.6328125) proc.memory.availableMB‚ 1310.87890625$ disk./.usageGB‚2720.133255004883 disk.sda1.out‚0 disk.sda2.in‚ 128.6328125 disk.sdb2.out‚1700.755859375 disk.sda.out‚1700.755859375 disk.md0p1.out‚ 1656.265625* proc.memory.percent‚18.171024322509766 cpu‚25.215732253204823 disk.sda1.in‚0 disk.nvme0n1.in‚ 7537.84375 _’Ç”­'‚(:‰ ÑÇëÔ†¹á gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãØäú’®'‚(:‡ àÇëÔÝ·²à gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¹4o¯'‚(:ä àÇëÔÄǃá disk.sda2.out‚ 1702.6640625 disk.sdb2.in‚ 70.9609375 disk.sda.in‚ 128.6328125 disk.sda1.in‚0 disk.md0.out‚ 1658.12109375$ disk./.usageGB‚2720.263900756836 disk.dm-0.in‚ 8055.95703125 disk.nvme1n1.in‚ 518.11328125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.4033203125 disk.nvme0n1.in‚ 7537.84375# disk.nvme0n1.out‚135672.0390625 network.recv‚ 3.706606e+06 disk.rbd0.in‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.21550532887717 proc.cpu.threads‚17 disk.dm-0.out‚189126.2734375 disk.md0p1.in‚ 199.59375# memory_percent‚91.9996976852417 network.sent‚ 2.4656336e+07) proc.memory.availableMB‚ 1310.76953125* proc.memory.percent‚18.171024322509766 disk.sda2.in‚ 128.6328125 disk.sdb.in‚ 70.9609375 disk.sdb2.out‚ 1702.6640625 disk.md0.in‚ 199.59375) disk./.usagePercent‚76.05971718405273 disk.sdb.out‚ 1702.6640625 disk.md0p1.out‚ 1658.12109375 disk.sdb1.in‚0 disk.sda.out‚ 1702.6640625! disk.nvme1n1.out‚ 53454.234375 disk.rbd0.out‚ 3.44189453125 disk.sda1.out‚0‚wšë4°'/ _wandbruntime‚11162  _runtime‚11162Û˜š‘ʱ'Ä _step‚761 ' lr-AdamW/pg1‚0.00011250827936133227 ' lr-AdamW/pg2‚0.00011250827936133227 trainer/global_step‚9299 % _timestamp‚1.7885357791807287e+09 _runtime‚11162.7162019875[³Ï²'É trainer/global_step‚9299 % _timestamp‚1.7885357791807287e+09 _runtime‚11162.716201987 _step‚761 ' lr-AdamW/pg1‚0.00011250827936133227 ' lr-AdamW/pg2‚0.00011250827936133227ùve¨n4³'/ _wandbruntime‚11162  _runtime‚111627ç"(´'P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793659362252800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172762649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚863934056966656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116428390 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚783010414956032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10648037325312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10648037325312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚875052781087232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚767731599351535 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10609131065371 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚875052519976448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11118724120576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚783010414956032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚11162.717298272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11118486949888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793658434278400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 194323708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334257 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10608223090367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885357791816952e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚757123376261168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116428392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235468133 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235441789 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793658434278400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚762xyG (µ'†P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429765497 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚783010396953088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429791843 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116428392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235468133 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235441789 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793658434278400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚762 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793659362252800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172762649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚863934056966656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116428390 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚783010414956032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10648037325312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10648037325312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚875052781087232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚767731599351535 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10609131065371 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚875052519976448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11118724120576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚783010414956032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚11162.717298272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11118486949888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793658434278400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 194323708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334257 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10608223090367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885357791816952e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚757123376261168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116440457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194343870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194343868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757202410895072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚9299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793742102812160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783092016038912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10649158785536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚783092034041856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172779850 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235492990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172780460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56340003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429836860 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10649158785536 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116440455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚783092034041856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10650068770304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429810488 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235466620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚864024282671104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885357801414604e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429836860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235466620 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757202393855712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10609340363645 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56339395 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚763 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235492990 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194343870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11119657311232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11163.677436772 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194343868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚767811734219357 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144hf¹‚'¸'N PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116440457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194343870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194343868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757202410895072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚9299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793742102812160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783092016038912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10649158785536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚783092034041856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172779850 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235492990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172780460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56340003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429836860 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10649158785536 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116440455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚783092034041856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10650068770304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429810488 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235466620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚864024282671104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885357801414604e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429836860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235466620 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757202393855712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10609340363645 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56339395 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚763 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235492990 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194343870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11119657311232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11163.677436772 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194343868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚767811734219357 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144ûÄ´;.4¹'/ _wandbruntime‚11163  _runtime‚11163ôˆïXº'Ò $loss/total_loss‚38.2680508984082 0loss/FRt2/assignment_loss‚3.3977276222272352 /loss/FRt1/detection_loss‚2.9626831720615256 0loss/FRt3/assignment_loss‚0.8826813893531685 2loss/SRqqt1/assignment_loss‚1.1993316478770355 0loss/FBt2/assignment_loss‚0.1597524943922915 .loss/FRt2/detection_loss‚2.167100135076246 1loss/FBt1/assignment_loss‚0.01912772529594804 1loss/SRqqt1/detection_loss‚2.4433670228925246 0loss/SRqqt2/detection_loss‚1.810558553861112 1loss/SRqqt4/detection_loss‚2.1637407082158164 0loss/SRqqt3/assignment_loss‚0.65593167537603 2loss/SRqqt4/assignment_loss‚1.4028558480943478 /loss/FRt4/detection_loss‚2.1612746450783953 1loss/SRqqt3/detection_loss‚0.9385654645100716 /loss/FBt1/detection_loss‚2.3252631306648253 /loss/FBt3/detection_loss‚0.6032767007332843 _runtime‚11163.678943339 0loss/FRt4/assignment_loss‚2.6750743970984505 0loss/FBt4/assignment_loss‚0.4177767008254442 .loss/FBt2/detection_loss‚2.253934781659733 /loss/FBt4/detection_loss‚2.1946165515826297 epoch‚7 % _timestamp‚1.7885357801436698e+09 _step‚764 .loss/FRt3/detection_loss‚1.173980869213814 /loss/FRt1/assignment_loss‚2.889561488710601 0loss/SRqqt2/assignment_loss‚1.26799859866804 1loss/FBt3/assignment_loss‚0.10186957493963172 trainer/global_step‚9299#¾ó!]»'× /loss/FRt1/assignment_loss‚2.889561488710601 0loss/SRqqt2/assignment_loss‚1.26799859866804 1loss/FBt3/assignment_loss‚0.10186957493963172 trainer/global_step‚9299 $loss/total_loss‚38.2680508984082 0loss/FRt2/assignment_loss‚3.3977276222272352 /loss/FRt1/detection_loss‚2.9626831720615256 0loss/FRt3/assignment_loss‚0.8826813893531685 2loss/SRqqt1/assignment_loss‚1.1993316478770355 0loss/FBt2/assignment_loss‚0.1597524943922915 .loss/FRt2/detection_loss‚2.167100135076246 1loss/FBt1/assignment_loss‚0.01912772529594804 1loss/SRqqt1/detection_loss‚2.4433670228925246 0loss/SRqqt2/detection_loss‚1.810558553861112 1loss/SRqqt4/detection_loss‚2.1637407082158164 0loss/SRqqt3/assignment_loss‚0.65593167537603 2loss/SRqqt4/assignment_loss‚1.4028558480943478 /loss/FRt4/detection_loss‚2.1612746450783953 1loss/SRqqt3/detection_loss‚0.9385654645100716 /loss/FBt1/detection_loss‚2.3252631306648253 /loss/FBt3/detection_loss‚0.6032767007332843 _runtime‚11163.678943339 0loss/FRt4/assignment_loss‚2.6750743970984505 0loss/FBt4/assignment_loss‚0.4177767008254442 .loss/FBt2/detection_loss‚2.253934781659733 /loss/FBt4/detection_loss‚2.1946165515826297 epoch‚7 % _timestamp‚1.7885357801436698e+09 _step‚764 .loss/FRt3/detection_loss‚1.173980869213814ü¢µ+B’¼'‚(:‡ ïÇëÔÌŸªá gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¿Ódq½'‚(:æ ïÇëÔÞûáá disk.sdb1.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06156881089036$ disk./.usageGB‚2720.330123901367# disk.nvme1n1.out‚53454.48828125 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.84375$ disk.nvme0n1.out‚135879.35546875 disk.sda2.in‚ 128.6328125 network.sent‚ 2.4674509e+07 disk.md0.out‚ 1659.546875 disk.sdb.in‚ 70.9609375 disk.sda.out‚1704.150390625 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.2152355313724 disk.md0.in‚ 199.59375 disk.sdb2.out‚1704.150390625 disk.sdb1.out‚0# memory_percent‚92.0006275177002 disk.sda.in‚ 128.6328125 network.recv‚ 3.710532e+06 disk.rbd2.out‚ 3.4033203125 disk.sda2.out‚1704.150390625 disk.rbd0.out‚ 3.4658203125 disk.dm-0.out‚ 189333.84375 disk.sdb.out‚1704.150390625 disk.nvme1n1.in‚ 518.11328125( proc.memory.availableMB‚ 1310.6171875 disk.sdb2.in‚ 70.9609375 disk.md0p1.in‚ 199.59375 disk.md0p1.out‚ 1659.546875 disk.rbd0.in‚0é·’¾'‚(:‡ þÇëÔœšŸà gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wO»²}¿'‚(:ò þÇëÔˆóÔà disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1661.015625 disk.sda.in‚ 128.6328125! disk.sda.out‚1705.66357421875# disk.nvme1n1.out‚53454.72265625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 3.46630859375# disk.nvme0n1.out‚136064.0546875$ disk./.usageGB‚2720.398136138916 disk.nvme0n1.in‚ 7537.84375! disk.dm-0.out‚189518.77734375 disk.md0p1.in‚ 199.59375 disk.rbd0.out‚ 3.4658203125 disk.sda1.in‚0( proc.memory.availableMB‚ 1310.6171875 disk.md0.in‚ 199.59375" disk.sdb2.out‚1705.66357421875) disk./.usagePercent‚76.06347046151755 disk.dm-0.in‚ 8055.95703125 disk.rbd2.in‚0 cpu‚25.215019694433142 network.recv‚ 3.717387e+06 disk.sdb.in‚ 70.9609375! disk.sdb.out‚1705.66357421875 disk.sdb2.in‚ 70.9609375 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.out‚ 1661.015625 disk.sda1.out‚0 proc.cpu.threads‚17" disk.sda2.out‚1705.66357421875# memory_percent‚92.0006275177002 network.sent‚ 2.477139e+07>0ΞÀ'‚(:“ ÈëÔΊÜà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yŠEÏÁ'‚(:ô ÈëÔþ¦úà% disk./.usageGB‚2720.4023094177246 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7537.84375$ disk.nvme0n1.out‚136229.02734375" disk.sda2.out‚1707.62646484375! disk.sdb.out‚1707.62646484375 disk.rbd2.in‚0 disk.rbd0.out‚ 3.52880859375 disk.sda1.out‚0 cpu‚25.214775658538855 proc.cpu.threads‚17 disk.nvme1n1.in‚ 518.11328125 disk.nvme1n1.out‚ 53454.875 disk.sda2.in‚ 128.6328125 disk.md0.out‚ 1662.91796875( proc.memory.availableMB‚ 1310.6015625 disk.md0p1.out‚ 1662.91796875 disk.dm-0.in‚ 8055.95703125! disk.dm-0.out‚189683.90234375 network.recv‚ 3.729049e+06 disk.md0p1.in‚ 199.59375 disk.rbd0.in‚0 disk.rbd2.out‚ 3.46630859375 disk.sdb.in‚ 70.9609375 disk.sdb2.in‚ 70.9609375! disk.sda.out‚1707.62646484375! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.478106e+07( disk./.usagePercent‚76.0635871481397 disk.sda1.in‚0 disk.md0.in‚ 199.59375" disk.sdb2.out‚1707.62646484375$ memory_percent‚92.00072288513184* proc.memory.percent‚18.171024322509766 disk.sdb1.in‚0 disk.sda.in‚ 128.6328125úî_Z4Â'/ _wandbruntime‚11219  _runtime‚11219_päÇÃ'Á & lr-AdamW/pg1‚0.0001087242377467142 & lr-AdamW/pg2‚0.0001087242377467142 trainer/global_step‚9349 $ _timestamp‚1.788535835486496e+09 _runtime‚11219.022089127 _step‚765a¡ÑÌÄ'Æ _runtime‚11219.022089127 _step‚765 & lr-AdamW/pg1‚0.0001087242377467142 & lr-AdamW/pg2‚0.0001087242377467142 trainer/global_step‚9349 $ _timestamp‚1.788535835486496e+09ýÀ>tA4Å'/ _wandbruntime‚11219  _runtime‚11219w¼0ã;Æ'P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10704124750336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236686691 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10664100345475 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚9349 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195331708 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797796403969024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 432044745 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56624119 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11177539679744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚797795475994624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879622631243776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117031640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195331710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117031642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10665008320479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚11219.023542783 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787091369247232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868445342189056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.smalTÃÔÌl_pool.allocated‚56624748 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173655759 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885358354876132e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚771738356333843 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 _step‚766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236686691 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10705034721792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 432018399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797796403969024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787091369247232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761074273027728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879622881868800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787091351244288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868445318248960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797795475994624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10704124750336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10705034721792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195331708 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ù»‰G (Ç'†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236686691 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10705034721792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 432018399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797796403969024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787091369247232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761074273027728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879622881868800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787091351244288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868445318248960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797795475994624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10704124750336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10705034721792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195331708 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10704124750336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236686691 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10664100345475 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚9349 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195331708 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797796403969024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 432044745 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56624119 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11177539679744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚797795475994624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879622631243776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117031640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195331710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117031642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10665008320479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚11219.023542783 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787091369247232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868445342189056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56624748 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173655759 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885358354876132e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚771738356333843 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 _step‚766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2þCŒFÿ”È'‚(:‰ œÈëÔæŸøà gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡3SñzÉ'‚(:ï œÈëÔ–è…ã$ disk./.usageGB‚2720.653064727783 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 7537.84375# disk.nvme0n1.out‚136361.1796875 proc.cpu.threads‚17) disk./.usagePercent‚76.07059837155819! disk.sda2.out‚1709.8818359375 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sda.out‚1709.8818359375 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.dm-0.in‚ 8055.95703125 disk.sda.in‚ 133.16796875 disk.md0p1.out‚1665.125( proc.memory.availableMB‚ 1310.3203125! proc.memory.rssMB‚ 2977.140625 disk.rbd2.out‚ 3.529296875 disk.md0.out‚1665.125$ memory_percent‚92.00243949890137! disk.dm-0.out‚189816.36328125! disk.sdb2.out‚1709.8818359375 cpu‚25.214546052182513# disk.nvme1n1.out‚53455.18359375 network.sent‚ 2.4797947e+07 network.recv‚ 3.733691e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.rbd0.out‚ 3.52880859375 disk.sda1.out‚0 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1709.8818359375* proc.memory.percent‚18.171024322509766衼l4Ê'/ _wandbruntime‚11220  _runtime‚11220‰#G'Ë'‹N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚771819416216237 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚761153290622272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195351868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9349 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚761153273582912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10705246210560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56629983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787172988333056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173674329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879714283667968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432089762 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885358366458988e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771818491201665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195351868 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10706156195328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797879144528384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117043707 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195351870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚868535567893504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10705246210560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236737892 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11220.183351209 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797878216540672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787172988333056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879714037250560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10666125593965 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚797879144528384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173673688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117043705 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10665217618753 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11178493297152 _step‚767 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10705246210560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56629983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787172988333056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173674329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879714283667968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432089762 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885358366458988e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771818491201665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195351868 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10706156195328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797879144528384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117043707 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195351870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚868535567893504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10705246210560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236737892 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ¹šrXJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11220.183351209 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797878216540672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787172988333056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879714037250560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10666125593965 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚797879144528384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173673688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117043705 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10665217618753 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11178493297152 _step‚767 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÿÒq—w4Í'/ _wandbruntime‚11220  _runtime‚11220Zzó\Î'Ö 2loss/SRqqt1/assignment_loss‚1.2341468296945095 2loss/FBt3/assignment_loss‚0.046649350235220884 .loss/FRt1/detection_loss‚2.840571448802948 /loss/FRt4/detection_loss‚2.5316695280605925 _runtime‚11220.18569653 0loss/FRt3/assignment_loss‚0.9735877675669534 /loss/FRt4/assignment_loss‚2.843871346116066 1loss/FBt1/assignment_loss‚0.13364993874009606 0loss/SRqqt1/detection_loss‚2.366790840195285 trainer/global_step‚9349 % _timestamp‚1.7885358366501606e+09 2loss/SRqqt4/assignment_loss‚1.5414145123691132 0loss/FBt2/assignment_loss‚0.0954274504025913 1loss/SRqqt4/detection_loss‚1.8692821837388551 /loss/FRt1/assignment_loss‚2.096794829368591 2loss/SRqqt3/assignment_loss‚0.8565673953806511 1loss/FBt4/assignment_loss‚0.47604626870309275 1loss/SRqqt2/detection_loss‚2.5709168034972567 1loss/SRqqt3/detection_loss‚0.8620982485437634 /loss/FBt1/detection_loss‚3.0699488520622253 /loss/FRt2/assignment_loss‚3.318855957343028 /loss/FRt2/detection_loss‚2.5723176999941084 /loss/FBt2/detection_loss‚1.9545741558074952 /loss/FBt3/detection_loss‚0.5504179547255438 /loss/FBt4/detection_loss‚1.8863107774938856 1loss/SRqqt2/assignment_loss‚1.345200198984733 .loss/FRt3/detection_loss‚1.592134992991175 %loss/total_loss‚39.62924533081778 epoch‚7 _step‚768äçe±aÏ'Û /loss/FBt3/detection_loss‚0.5504179547255438 /loss/FBt4/detection_loss‚1.8863107774938856 1loss/SRqqt2/assignment_loss‚1.345200198984733 .loss/FRt3/detection_loss‚1.592134992991175 %loss/total_loss‚39.62924533081778 epoch‚7 _step‚768 2loss/SRqqt1/assignment_loss‚1.2341468296945095 2loss/FBt3/assignment_loss‚0.046649350235220884 .loss/FRt1/detection_loss‚2.840571448802948 /loss/FRt4/detection_loss‚2.5316695280605925 _runtime‚11220.18569653 0loss/FRt3/assignment_loss‚0.9735877675669534 /loss/FRt4/assignment_loss‚2.843871346116066 1loss/FBt1/assignment_loss‚0.13364993874009606 0loss/SRqqt1/detection_loss‚2.366790840195285 trainer/global_step‚9349 % _timestamp‚1.7885358366501606e+09 2loss/SRqqt4/assignment_loss‚1.5414145123691132 0loss/FBt2/assignment_loss‚0.0954274504025913 1loss/SRqqt4/detection_loss‚1.8692821837388551 /loss/FRt1/assignment_loss‚2.096794829368591 2loss/SRqqt3/assignment_loss‚0.8565673953806511 1loss/FBt4/assignment_loss‚0.47604626870309275 1loss/SRqqt2/detection_loss‚2.5709168034972567 1loss/SRqqt3/detection_loss‚0.8620982485437634 /loss/FBt1/detection_loss‚3.0699488520622253 /loss/FRt2/assignment_loss‚3.318855957343028 /loss/FRt2/detection_loss‚2.5723176999941084 /loss/FBt2/detection_loss‚1.9545741558074952€Ãf!>|Ð'‚(:ñ «ÈëÔÏöôâ disk.sda2.in‚ 133.16796875 cpu‚25.214305244905045 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 190121.578125 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1711.50341796875 disk.sda.in‚ 133.16796875$ memory_percent‚92.00112819671631) proc.memory.availableMB‚ 1310.53515625$ disk./.usageGB‚2720.903263092041 proc.cpu.threads‚17 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 7537.84375 disk.md0.out‚ 1666.6953125 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625' disk./.usagePercent‚76.077594022539 disk.rbd2.in‚0! disk.sdb.out‚1711.50341796875 disk.nvme1n1.in‚ 518.11328125* proc.memory.percent‚18.171024322509766 disk.sda1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1711.50341796875 disk.md0.in‚ 208.765625 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0" disk.nvme0n1.out‚ 136666.140625! disk.sda.out‚1711.50341796875 disk.nvme1n1.out‚ 53455.4375 disk.md0p1.out‚ 1666.6953125 network.sent‚ 2.4881097e+07 network.recv‚ 3.740195e+06 disk.rbd2.out‚ 3.529296875£š–Ë“Ñ'‚(:ˆ «ÈëÔƒÏá gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈkDï”Ò'‚(:‰ ºÈëÔâÖøÞ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚153.509' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.37725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÐ4Ð…Ó'‚(:ú ºÈëÔ¢îà disk.sdb.in‚ 75.59765625 disk.md0.out‚ 1668.11328125) disk./.usagePercent‚76.08106454293316# disk./.usageGB‚2721.02738571167 disk.rbd0.in‚0$ memory_percent‚92.00365543365479) proc.memory.availableMB‚ 1310.12109375 cpu‚25.21407745873902 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 133.16796875! disk.sdb.out‚1712.96826171875 disk.md0p1.out‚ 1668.11328125 network.recv‚ 3.746394e+06 disk.sda1.out‚0 disk.dm-0.in‚ 8055.95703125! disk.dm-0.out‚190485.80859375 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.11328125 network.sent‚ 2.4978103e+07 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0# disk.nvme0n1.out‚136913.5703125! disk.sda.out‚1712.96826171875# disk.nvme1n1.out‚53572.23828125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7537.84375 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1712.96826171875 disk.md0p1.in‚ 208.765625* proc.memory.percent‚18.171024322509766 disk.rbd2.in‚0" disk.sda2.out‚1712.96826171875! proc.memory.rssMB‚ 2977.140625b³`H”Ô'‚(:‰ ÉÈëÔ˜ý ß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öa¶…Õ'‚(:ú ÉÈëÔèíîà proc.cpu.threads‚17 network.sent‚ 2.4984392e+07 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8055.95703125! disk.dm-0.out‚190681.43359375 disk.nvme1n1.out‚ 53572.40625 disk.sdb1.out‚0 disk.sda1.in‚0$ memory_percent‚92.00236797332764$ disk.nvme0n1.out‚137109.02734375! disk.sda.out‚1715.00830078125 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1310.33203125* proc.memory.percent‚18.171024322509766 disk.nvme0n1.in‚ 7537.84375 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1715.00830078125 disk.md0p1.in‚ 208.765625 network.recv‚ 3.751323e+06! disk.sdb.out‚1715.00830078125 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.591796875 disk.rbd2.out‚ 3.529296875! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.09160516335956$ disk./.usageGB‚2721.404369354248 disk.md0.out‚ 1670.08984375 disk.md0p1.out‚ 1670.08984375 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1715.00830078125 disk.md0.in‚ 208.765625 disk.rbd2.in‚0 cpu‚25.213856084349516vB³)4Ö'/ _wandbruntime‚11276  _runtime‚11276Y‡Ê×'Ä trainer/global_step‚9399 % _timestamp‚1.7885358933007345e+09 _runtime‚11276.836338437 _step‚769 ' lr-AdamW/pg1‚0.00010499379112077364 ' lr-AdamW/pg2‚0.00010499379112077364–%6’ÏØ'É _step‚769 ' lr-AdamW/pg1‚0.00010499379112077364 ' lr-AdamW/pg2‚0.00010499379112077364 trainer/global_step‚9399 % _timestamp‚1.7885358933007345e+09 _runtime‚11276.836338437‘ÚÕ4Ù'/ _wandbruntime‚11276  _runtime‚11276÷; p(Ú'P OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884192776322560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791172305535488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775745113316151 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10720885575587 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801932517710848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434297647 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174549187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237931593 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11236148911104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚770 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196339708 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10760212175360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885358933018732e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196339710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56914297 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚801932517710848 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117634890 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791172323538432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10761122146816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801933445685248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196339708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚9399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791172305535488Ë (Û'†P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚872956627411456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56914915 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚884192525697536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚791172323538432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237957937 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237931593 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11235922226176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚801933445685248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434271301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 237957937 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚872956603471360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚765025152754928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196339710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56914297 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚801932517710848 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117634890 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791172323538432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10761122146816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801933445685248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196339708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚9399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791172305535488 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884192776322560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791172305535488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775745113316151 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10720885575587 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801932517710848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434297647 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174549187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237931593 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11236148911104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚770 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196339708 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10760212175360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885358933018732e+09 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ú^‘{& OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237982794 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174567587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56920630 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196359870 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11237317327872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775826173198545 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10721094873861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434342664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10762243620352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196359868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚765104153310112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚873046829175808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802016186244608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802016186244608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117646955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10722002849073 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791253924621312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196359868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237956424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11237086461952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791253942624256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802015258256896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196359870 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791253942624256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10761333635584 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117646957 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775825248183973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237982794 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237956424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚791253924621312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56920000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10761333635584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434342664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174566955 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765104170349472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚771Õf1'Þ'N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434342664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174566955 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765104170349472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚771 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237982794 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174567587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56920630 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196359870 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11237317327872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775826173198545 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10721094873861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434342664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10762243620352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196359868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚765104153310112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚873046829175808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802016186244608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802016186244608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117646955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10722002849073 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791253924621312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196359868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237956424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11237086461952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791253942624256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802015258256896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196359870 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791253942624256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10761333635584 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117646957 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775825248183973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237982794 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237956424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚791253924621312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56920000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10761333635584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231ƒ0ì4ß'/ _wandbruntime‚11277  _runtime‚11277Üì¾Zà'Ô 0loss/FRt2/assignment_loss‚3.1741275872503008 1loss/FBt2/assignment_loss‚0.09299649292046928 .loss/FRt2/detection_loss‚2.437057783603502 /loss/SRqqt2/detection_loss‚2.27545019827391 /loss/FBt3/detection_loss‚0.7972211684981311 % _timestamp‚1.7885358944314203e+09 /loss/FRt1/assignment_loss‚2.775401044636965 2loss/SRqqt2/assignment_loss‚1.2160472192979221 /loss/FBt1/detection_loss‚2.6950852970282235 .loss/FBt2/detection_loss‚2.637464059723748 2loss/SRqqt1/assignment_loss‚1.7030666682582636 2loss/SRqqt3/assignment_loss‚0.9205517287305354 1loss/FBt4/assignment_loss‚0.10679350300947969 /loss/FRt4/detection_loss‚2.2156633167312694 0loss/SRqqt1/detection_loss‚2.195171888058002 1loss/SRqqt3/detection_loss‚0.7966194249406133 _step‚772 1loss/SRqqt4/assignment_loss‚1.400719726859939 3loss/FBt1/assignment_loss‚0.0026333770180523666 1loss/FBt3/assignment_loss‚0.17825392949475943 %loss/total_loss‚39.68864522959507 .loss/FRt1/detection_loss‚2.593564825132489 .loss/FBt4/detection_loss‚2.175456499059995 0loss/FRt4/assignment_loss‚2.8316637036892085 epoch‚7 _runtime‚11277.966987144 /loss/FRt3/assignment_loss‚1.221558740426754 .loss/FRt3/detection_loss‚1.389074481863143 0loss/SRqqt4/detection_loss‚1.857002565089394 trainer/global_step‚9399{â_á'Ù /loss/FRt1/assignment_loss‚2.775401044636965 2loss/SRqqt2/assignment_loss‚1.2160472192979221 /loss/FBt1/detection_loss‚2.6950852970282235 .loss/FBt2/detection_loss‚2.637464059723748 2loss/SRqqt1/assignment_loss‚1.7030666682582636 2loss/SRqqt3/assignment_loss‚0.9205517287305354 1loss/FBt4/assignment_loss‚0.10679350300947969 /loss/FRt4/detection_loss‚2.2156633167312694 0loss/SRqqt1/detection_loss‚2.195171888058002 1loss/SRqqt3/detection_loss‚0.7966194249406133 _step‚772 1loss/SRqqt4/assignment_loss‚1.400719726859939 3loss/FBt1/assignment_loss‚0.0026333770180523666 1loss/FBt3/assignment_loss‚0.17825392949475943 %loss/total_loss‚39.68864522959507 .loss/FRt1/detection_loss‚2.593564825132489 .loss/FBt4/detection_loss‚2.175456499059995 0loss/FRt4/assignment_loss‚2.8316637036892085 epoch‚7 _runtime‚11277.966987144 /loss/FRt3/assignment_loss‚1.221558740426754 .loss/FRt3/detection_loss‚1.389074481863143 0loss/SRqqt4/detection_loss‚1.857002565089394 trainer/global_step‚9399 0loss/FRt2/assignment_loss‚3.1741275872503008 1loss/FBt2/assignment_loss‚0.09299649292046928 .loss/FRt2/detection_loss‚2.437057783603502 /loss/SRqqt2/detection_loss‚2.27545019827391 /loss/FBt3/detection_loss‚0.7972211684981311 % _timestamp‚1.7885358944314203e+09„)¯”â'‚(:‰ ØÈëÔÛ’à gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼R—„ã'‚(:ù ØÈëÔÅÿÑà network.sent‚ 2.4990562e+07 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7537.84375" disk.sdb2.out‚1716.41552734375 disk.sda.in‚ 133.16796875# disk.nvme1n1.out‚53572.66796875! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sda1.in‚0 disk.rbd2.out‚ 3.529296875 disk.md0.out‚ 1671.45703125 cpu‚25.21361748023714 proc.cpu.threads‚17 network.recv‚ 3.755482e+06) disk./.usagePercent‚76.08379579917921 disk.md0p1.out‚ 1671.45703125 disk.rbd0.out‚ 3.591796875 disk.sda1.out‚0$ disk.nvme0n1.out‚137251.48046875 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 518.11328125# memory_percent‚92.0041561126709 disk.sda2.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb2.in‚ 75.59765625! disk.sda.out‚1716.41552734375$ disk./.usageGB‚2721.125068664551 disk.sdb1.out‚0" disk.sda2.out‚1716.41552734375 disk.dm-0.out‚190824.1484375! disk.sdb.out‚1716.41552734375( proc.memory.availableMB‚ 1310.0390625 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 8055.95703125•¿ä„ä'‚(:ù çÈëÔÝØåß disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 133.16796875$ memory_percent‚92.00184345245361 disk.sda.in‚ 133.16796875! disk.nvme1n1.out‚ 53572.921875 disk.dm-0.in‚ 8055.95703125 disk.md0.out‚ 1673.13671875 disk.dm-0.out‚191047.1328125 disk.sdb.in‚ 75.59765625) proc.memory.availableMB‚ 1310.41796875 cpu‚25.21340463576048 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0 disk.rbd2.in‚0! disk.sdb.out‚1718.14892578125" disk.sdb2.out‚1718.14892578125 proc.cpu.threads‚17 network.sent‚ 2.5097966e+07 network.recv‚ 3.762285e+06% disk./.usageGB‚2721.1504859924316 disk.nvme1n1.in‚ 518.11328125 disk.sdb2.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.md0p1.in‚ 208.765625 disk.rbd2.out‚ 3.529296875 disk.sdb1.out‚0) disk./.usagePercent‚76.08450647830473 disk.sdb1.in‚0# disk.nvme0n1.out‚137474.2109375" disk.sda2.out‚1718.14892578125! disk.sda.out‚1718.14892578125 disk.md0.in‚ 208.765625 disk.md0p1.out‚ 1673.13671875 disk.rbd0.in‚0sÚƒX”å'‚(:‰ çÈëÔÕ‘®à gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°»ˆñ”æ'‚(:‰ öÈëÔ³›ôÞ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•SÒ„ç'‚(:ù öÈëÔ×ßáà$ disk./.usageGB‚2721.213176727295 disk.rbd0.in‚0 disk.rbd2.out‚ 3.529296875 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875! disk.sda.out‚1720.05126953125 proc.cpu.threads‚17 network.recv‚ 3.766048e+06 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.out‚ 1674.9765625 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sda2.out‚1720.05126953125 disk.md0.out‚ 1674.9765625$ memory_percent‚92.00279712677002 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 7537.84375 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 75.59765625) proc.memory.availableMB‚ 1310.26171875! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.21318392283579 disk.sda1.out‚0! disk.dm-0.out‚191144.07421875 network.sent‚ 2.5103661e+07 disk.rbd2.in‚0$ disk.nvme0n1.out‚137571.33203125" disk.sdb2.out‚1720.05126953125) disk./.usagePercent‚76.08625933748962" disk.nvme1n1.out‚ 53573.3671875 disk.md0.in‚ 208.765625! disk.sdb.out‚1720.05126953125 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875dÀ¶”è'‚(:‰ …ÉëÔš¥¨ß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë[vùué'‚(:ê …ÉëÔáÿÐà# disk.nvme0n1.out‚137737.8046875 disk.md0.out‚ 1676.64453125 disk.sda.in‚ 133.16796875 disk.sda.out‚ 1721.7890625# disk.nvme1n1.out‚53573.54296875 disk.md0p1.in‚ 208.765625 disk.sda2.out‚ 1721.7890625 disk.dm-0.in‚ 8055.95703125! disk.dm-0.out‚191311.34765625( proc.memory.availableMB‚ 1310.0390625 disk.sdb.out‚ 1721.7890625# memory_percent‚92.0041561126709! proc.memory.rssMB‚ 2977.140625% disk./.usageGB‚2721.4006423950195 disk.sdb2.out‚ 1721.7890625 disk.rbd0.out‚ 3.591796875 cpu‚25.21294680851715) disk./.usagePercent‚76.09150095601969 disk.sdb2.in‚ 75.59765625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.md0.in‚ 208.765625 network.sent‚ 2.5109873e+07 disk.sdb.in‚ 75.59765625 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875* proc.memory.percent‚18.171024322509766 disk.nvme1n1.in‚ 518.11328125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 network.recv‚ 3.770795e+06 disk.md0p1.out‚ 1676.64453125 disk.sda1.in‚0 disk.nvme0n1.in‚ 7537.84375Þ·BÆ4ê'/ _wandbruntime‚11334  _runtime‚11334’âÀÊë'Ä ' lr-AdamW/pg1‚0.00010131771690299627 ' lr-AdamW/pg2‚0.00010131771690299627 trainer/global_step‚9449 % _timestamp‚1.7885359510826757e+09 _runtime‚11334.618116002 _step‚773™PæÏì'É trainer/global_step‚9449 % _timestamp‚1.7885359510826757e+09 _runtime‚11334.618116002 _step‚773 ' lr-AdamW/pg1‚0.00010131771690299627 ' lr-AdamW/pg2‚0.00010131771690299627…ägiÝ4í'/ _wandbruntime‚11334  _runtime‚11334+sز î'P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚795253259826688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239176495 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10817209571840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175442036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57203896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197347708 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10776762830695 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10816299600384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239202839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175442646 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885359510836203e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚806069559427072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118238142 ]ODeviceStatsMonitor.{:ɈUon_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚774 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436550549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888762733664256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚11334.619210657 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚888762474650624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚877467912633856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚768976032482128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197347708 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚795253277829632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436524203 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11294821030400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚768976015442768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806069559427072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚806070487401472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10817209571840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚795253277829632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877467888693760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10816299600384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239202839 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239176495 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197347710 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57204504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10775854855691 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚779751870298459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806069559427072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚806070487401472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10817209571840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚795253277829632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877467888693760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10816299600384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239202839 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239176495 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197347710 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57204504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10775854855691 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚779751870298459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175442036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57203896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197347708 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10776762830695 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10816299600384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239202839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175442646 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885359510836203e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚806069559427072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118238142 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚774 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436550549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888762733664256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚11334.619210657†—g%“4ð'/ _wandbruntime‚11335  _runtime‚11335¢$'ñ'‹N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197367870 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚795334896915456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚775 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197367868 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175459833 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888853867767296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436595566 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197367870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚806153227960832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118250205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779832005166281 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888854122573312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11295753369088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚769055033037312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10817421060608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877558138338304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10777880104181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175460473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10817421060608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795334878912512 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877558114398208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10776972128969 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885359521569083e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436569194 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239227696 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118250207 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11335.694057491 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436595566 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197367868 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239201326 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚806153227960832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 DŒ7ò'N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795334878912512 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877558114398208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10776972128969 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885359521569083e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436569194 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239227696 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118250207 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11335.694057491 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436595566 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197367868 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239201326 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚806153227960832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DevnË©ëßiceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197367870 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚795334896915456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚775 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197367868 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175459833 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888853867767296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436595566 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197367870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚806153227960832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118250205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779832005166281 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888854122573312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11295753369088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚769055033037312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10817421060608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877558138338304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10777880104181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175460473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10817421060608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615‡cu4ó'/ _wandbruntime‚11335  _runtime‚11335dÉ>ýXô'Ò /loss/FRt2/assignment_loss‚2.491657493930114 2loss/SRqqt2/assignment_loss‚1.3150142945349217 1loss/FBt4/assignment_loss‚0.23143580347558934 /loss/FRt4/detection_loss‚2.2208735946227205 /loss/FRt4/assignment_loss‚2.458829542452639 1loss/FBt3/assignment_loss‚0.07735344219420505 .loss/FRt1/detection_loss‚2.733820776785574 /loss/FBt2/detection_loss‚1.9364995270967484 /loss/FBt3/detection_loss‚0.6357688013198121 2loss/SRqqt4/assignment_loss‚1.3172189007912363 /loss/FRt3/detection_loss‚1.4241645175261775 0loss/SRqqt4/detection_loss‚1.892141960206486 _runtime‚11335.696066835 /loss/FRt3/assignment_loss‚1.155892400637917 /loss/SRqqt2/detection_loss‚2.11776858993939 .loss/FBt1/detection_loss‚2.457278662919998 /loss/FBt4/detection_loss‚2.0231873790423074 _step‚776 /loss/FRt1/assignment_loss‚3.073731860806865 0loss/SRqqt1/detection_loss‚2.577847318649292 %loss/total_loss‚38.30994895847108 2loss/SRqqt1/assignment_loss‚1.5877790385484696 0loss/FBt1/assignment_loss‚0.3975226650713012 2loss/SRqqt3/assignment_loss‚0.9093425562605262 1loss/SRqqt3/detection_loss‚0.7779415844526795 trainer/global_step‚9449 0loss/FBt2/assignment_loss‚0.3007779580570055 /loss/FRt2/detection_loss‚2.1961002891491117 epoch‚7 % _timestamp‚1.7885359521599553e+09¶ÝËÕ]õ'× 2loss/SRqqt3/assignment_loss‚0.9093425562605262 1loss/SRqqt3/detection_loss‚0.7779415844526795 trainer/global_step‚9449 0loss/FBt2/assignment_loss‚0.3007779580570055 /loss/FRt2/detection_loss‚2.1961002891491117 epoch‚7 % _timestamp‚1.7885359521599553e+09 /loss/FRt2/assignment_loss‚2.491657493930114 2loss/SRqqt2/assignment_loss‚1.3150142945349217 1loss/FBt4/assignment_loss‚0.23143580347558934 /loss/FRt4/detection_loss‚2.2208735946227205 /loss/FRt4/assignment_loss‚2.458829542452639 1loss/FBt3/assignment_loss‚0.07735344219420505 .loss/FRt1/detection_loss‚2.733820776785574 /loss/FBt2/detection_loss‚1.9364995270967484 /loss/FBt3/detection_loss‚0.6357688013198121 2loss/SRqqt4/assignment_loss‚1.3172189007912363 /loss/FRt3/detection_loss‚1.4241645175261775 0loss/SRqqt4/detection_loss‚1.892141960206486 _runtime‚11335.696066835 /loss/FRt3/assignment_loss‚1.155892400637917 /loss/SRqqt2/detection_loss‚2.11776858993939 .loss/FBt1/detection_loss‚2.457278662919998 /loss/FBt4/detection_loss‚2.0231873790423074 _step‚776 /loss/FRt1/assignment_loss‚3.073731860806865 0loss/SRqqt1/detection_loss‚2.577847318649292 %loss/total_loss‚38.30994895847108 2loss/SRqqt1/assignment_loss‚1.5877790385484696 0loss/FBt1/assignment_loss‚0.3975226650713012ˆƒÞî”ö'‚(:‰ ”ÉëÔÀûáÞ gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V,Dl|÷'‚(:ñ ”ÉëÔº„§à disk.sdb.out‚1723.197265625 disk.sdb2.in‚ 75.59765625 disk.md0.in‚ 208.765625 disk.sda2.out‚1723.197265625# disk.nvme1n1.out‚53573.84765625) disk./.usagePercent‚76.09161102968883 disk.dm-0.in‚ 8055.95703125 disk.sdb2.out‚1723.197265625 disk.sda.out‚1723.197265625 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 133.16796875 network.recv‚ 3.77517e+06 disk.md0.out‚ 1678.0078125$ memory_percent‚92.00441837310791* proc.memory.percent‚18.171024322509766 cpu‚25.21268916698469 proc.cpu.threads‚17 disk.nvme1n1.in‚ 518.11328125! disk.dm-0.out‚191545.26171875 disk.sdb.in‚ 75.59765625 disk.md0p1.out‚ 1678.0078125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚137971.4140625! proc.memory.rssMB‚ 2977.140625 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.591796875 disk.rbd2.out‚ 3.529296875% disk./.usageGB‚2721.4045791625977 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 2.5128889e+07) proc.memory.availableMB‚ 1309.99609375Äk ”ø'‚(:‰ £ÉëԘݰà gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 j„ù'‚(:ù £ÉëÔ×ÙÓá" disk.nvme1n1.out‚ 53574.0390625 disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 133.16796875 disk.md0p1.in‚ 208.765625) proc.memory.availableMB‚ 1309.98828125 network.sent‚ 2.5223327e+07 disk.sda1.out‚0 disk.md0.out‚ 1679.4609375! disk.dm-0.out‚191907.47265625 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.out‚ 1679.4609375 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb.out‚1724.72998046875* proc.memory.percent‚18.171024322509766 disk.sdb.in‚ 75.59765625" disk.sdb2.out‚1724.72998046875 cpu‚25.21244655126356 disk.sda.in‚ 133.16796875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.84375$ disk.nvme0n1.out‚138333.43359375 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8055.95703125! proc.memory.rssMB‚ 2977.140625 network.recv‚ 3.78179e+06) disk./.usagePercent‚76.09160889647819 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875" disk.sda2.out‚1724.72998046875 proc.cpu.threads‚17% disk./.usageGB‚2721.4045028686523! disk.sda.out‚1724.72998046875$ memory_percent‚92.00446605682373¨ú”ú'‚(:‰ ²ÉëÔû¿úÞ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+"÷tû'‚(:é ²ÉëÔ݃†à disk.rbd2.in‚0 network.sent‚ 2.5229008e+07) disk./.usagePercent‚76.09510629532646 disk.dm-0.out‚ 192187.921875# disk.nvme1n1.out‚53690.69140625 disk.md0p1.out‚ 1681.578125$ memory_percent‚92.00465679168701 disk.sda2.in‚ 133.16796875 disk.sdb.out‚ 1726.91015625 disk.nvme1n1.in‚ 518.11328125 disk.sda1.in‚0! proc.memory.rssMB‚ 2977.140625$ disk./.usageGB‚2721.529586791992 disk.md0.out‚ 1681.578125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7537.84375$ disk.nvme0n1.out‚138497.23046875* proc.memory.percent‚18.171024322509766 disk.sdb2.in‚ 75.59765625 disk.rbd2.out‚ 3.529296875 disk.sda1.out‚0 network.recv‚ 3.78599e+06 disk.md0.in‚ 208.765625 disk.sdb1.in‚0) proc.memory.availableMB‚ 1309.95703125 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb2.out‚ 1726.91015625 disk.rbd0.out‚ 3.591796875 cpu‚25.21221814981707 disk.sda2.out‚ 1726.91015625 disk.sdb.in‚ 75.59765625 disk.sda.out‚ 1726.91015625Y¨I“ü'‚(:ˆ ÁÉëÔ·æµà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G©Äý'‚(:ô ÁÉëÔµþùá# disk.nvme1n1.out‚53690.86328125 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0# memory_percent‚92.0039176940918 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766 disk.sdb1.out‚0 network.sent‚ 2.5237493e+07$ disk./.usageGB‚2721.404499053955 disk.sda2.in‚ 133.16796875! disk.sda2.out‚1728.5185546875 disk.sdb2.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875 disk.dm-0.in‚ 8055.95703125 disk.sda.out‚1728.5185546875 disk.md0p1.out‚ 1683.13671875' proc.memory.availableMB‚ 1310.078125 disk.sdb.out‚1728.5185546875! disk.sdb2.out‚1728.5185546875 disk.sdb1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.790757e+06 disk.md0.out‚ 1683.13671875 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 518.11328125 cpu‚25.21199616890004) disk./.usagePercent‚76.09160878981766 disk.nvme0n1.in‚ 7537.84375# disk.nvme0n1.out‚138696.7265625 disk.md0.in‚ 208.765625! disk.dm-0.out‚192387.58984375 disk.sda.in‚ 133.16796875Ó ?4þ'/ _wandbruntime‚11392  _runtime‚11392íHÁÈÿ' & lr-AdamW/pg1‚9.769678118173923e-05 & lr-AdamW/pg2‚9.769678118173923e-05 trainer/global_step‚9499 % _timestamp‚1.7885360087702682e+09 _runtime‚11392.305939644 _step‚777×G|Í€(Ç & lr-AdamW/pg2‚9.769678118173923e-05 trainer/global_step‚9499 % _timestamp‚1.7885360087702682e+09 _runtime‚11392.305939644 _step‚777 & lr-AdamW/pg1‚9.769678118173923e-05‰Osz4(/ _wandbruntime‚11392  _runtime‚11392õ0Ô(‚(P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198355708 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚783758627280767 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚772926895169968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810207529117696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176336031 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240421397 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799334214117888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438803451 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57495244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10872387025408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚799334232120832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240447741 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10831732110799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438777105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚799334214117888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10873296996864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10873296996864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11353240369664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10832640085803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881979173916160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176336636 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810206601143296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885360087715564e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57494641 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚893332667008000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893332414285824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11353469151744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240421397 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚9499 _step‚778 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Ú|Ú#t ƒ(†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240421397 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799334214117888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438803451 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57495244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10872387025408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚799334232120832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240447741 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10831732110799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438777105 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚799334214117888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10873296996864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10873296996864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11353240369664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10832640085803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881979173916160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176336636 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810206601143296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885360087715564e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57494641 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚893332667008000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893332414285824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_b9ÉE¸˜ytes.small_pool.allocated‚11353469151744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240421397 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚9499 _step‚778 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198355708 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚783758627280767 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚772926895169968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810207529117696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176336031Šœ‰Æ4„(/ _wandbruntime‚11393  _runtime‚11393;ÂÖ{'…(‹N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 438848468 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799415833203712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176353893 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893424068997632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198375870 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438822096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚799415833203712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57500438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚799415851206656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882069399620608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773005912764512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198375868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚779 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10873508485632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810289341689344 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176354504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11354645436928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438848468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240472598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57501047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882069423560704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198375868 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚799415851206656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10874418470400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893423824677376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11354425056768 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240472598 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240446228 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚810290269677056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615´ÏGS'†(N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚810290269677056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 438848468 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799415833203712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176353893 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893424068997632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198375870 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438822096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚799415833203712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57500438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚799415851206656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882069399620608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773005912764512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198375868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚779 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10873508485632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810289341689344 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176354504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11354645436928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438848468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240472598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57501047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882069423560704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198375868 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚799415851206656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10874418470400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893423824677376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11354425056768 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240472598 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240446228 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144‹C4‡(/ _wandbruntime‚11393  _runtime‚11393‚"ŒœZˆ(Ô 0loss/FRt3/assignment_loss‚0.9841759649912516 0loss/FRt4/assignment_loss‚3.5108612992546777 1loss/FBt2/assignment_loss‚0.10200292981608072 .loss/FRt4/detection_loss‚2.319460447284986 epoch‚7 $ _timestamp‚1.788536009908063e+09 -loss/FRt1/detection_loss‚3.06005211353302 .loss/FRt2/detection_loss‚2.511823949731244 2loss/SRqqt1/assignment_loss‚1.6571770507704329 1loss/FBt1/assignment_loss‚0.21904816502046742 0loss/SRqqt1/detection_loss‚2.352730499373542 0loss/FBt3/detection_loss‚0.46045330246208144 %loss/total_loss‚42.97048401617213 1loss/SRqqt2/assignment_loss‚1.916562220454216 3loss/FBt4/assignment_loss‚9.536738048154803e-07 /loss/SRqqt3/detection_loss‚0.85589077989929 /loss/FBt1/detection_loss‚2.6671346766608104 /loss/FRt2/assignment_loss‚2.722462865614122 2loss/SRqqt4/assignment_loss‚1.0063620415088292 /loss/FRt3/detection_loss‚1.3630405794332425 1loss/SRqqt4/detection_loss‚1.9944624047021609 .loss/FBt2/detection_loss‚2.006172161210667 0loss/FRt1/assignment_loss‚3.6013375186920165 2loss/SRqqt3/assignment_loss‚1.0694267288602601 1loss/FBt3/assignment_loss‚0.12125633152860604 _runtime‚11393.443578034 _step‚780 1loss/SRqqt2/detection_loss‚2.0948183476924895 -loss/FBt4/detection_loss‚4.37377068400383 trainer/global_step‚94996q]Ê_‰(Ù 0loss/FRt4/assignment_loss‚3.5108612992546777 1loss/FBt2/assignment_loss‚0.10200292981608072 .loss/FRt4/detection_loss‚2.319460447284986 epoch‚7 $ _timestamp‚1.788536009908063e+09 -loss/FRt1/detection_loss‚3.06005211353302 .loss/FRt2/detection_loss‚2.511823949731244 2loss/SRqqt1/assignment_loss‚1.6571770507704329 1loss/FBt1/assignment_loss‚0.21904816502046742 0loss/SRqqt1/detection_loss‚2.352730499373542 0loss/FBt3/detection_loss‚0.46045330246208144 %loss/total_loss‚42.97048401617213 1loss/SRqqt2/assignment_loss‚1.916562220454216 3loss/FBt4/assignment_loss‚9.536738048154803e-07 /loss/SRqqt3/detection_loss‚0.85589077989929 /loss/FBt1/detection_loss‚2.6671346766608104 /loss/FRt2/assignment_loss‚2.722462865614122 2loss/SRqqt4/assignment_loss‚1.0063620415088292 /loss/FRt3/detection_loss‚1.3630405794332425 1loss/SRqqt4/detection_loss‚1.9944624047021609 .loss/FBt2/detection_loss‚2.006172161210667 0loss/FRt1/assignment_loss‚3.6013375186920165 2loss/SRqqt3/assignment_loss‚1.0694267288602601 1loss/FBt3/assignment_loss‚0.12125633152860604 _runtime‚11393.443578034 _step‚780 1loss/SRqqt2/detection_loss‚2.0948183476924895 -loss/FBt4/detection_loss‚4.37377068400383 trainer/global_step‚9499 0loss/FRt3/assignment_loss‚0.9841759649912516Œ&ÈÔ”Š(‚(:‰ ÐÉëÔÈÓìÞ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Võ¬W„‹(‚(:ù ÐÉëÔ™ïëá disk.sdb.in‚ 75.59765625" disk.sdb2.out‚1729.97314453125 disk.sda1.out‚0 disk.md0.out‚ 1684.54296875 disk.dm-0.in‚ 8055.95703125( proc.memory.availableMB‚ 1309.9140625! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.5241579e+07) disk./.usagePercent‚76.09161220295468 disk.rbd0.out‚ 3.591796875 disk.sdb2.in‚ 75.59765625 disk.sda2.in‚ 133.16796875! disk.dm-0.out‚192680.36328125 proc.cpu.threads‚17! disk.sda.out‚1729.97314453125 disk.md0p1.in‚ 208.765625 disk.rbd2.out‚ 3.529296875$ memory_percent‚92.00491905212402 cpu‚25.211762011473922 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.11328125 disk.md0p1.out‚ 1684.54296875 disk.sda1.in‚0* proc.memory.percent‚18.171024322509766% disk./.usageGB‚2721.4046211242676! disk.sdb.out‚1729.97314453125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7537.84375 network.recv‚ 3.795424e+06# disk.nvme1n1.out‚53691.17578125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.out‚ 138989.1875" disk.sda2.out‚1729.97314453125 disk.md0.in‚ 208.765625²¬W”Œ(‚(:‰ ßÉëÔž°©à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.017' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.25425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸK^(‚(:ö ßÉëÔšöùà disk.md0p1.in‚ 208.765625# disk.nvme0n1.out‚139218.5546875! disk.sdb.out‚1731.93408203125" disk.sdb2.out‚1731.93408203125 cpu‚25.211532074498457 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1731.93408203125* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 133.16796875 disk.md0.out‚ 1686.44140625 disk.sdb.in‚ 75.59765625 network.recv‚ 3.801613e+06 disk.md0p1.out‚ 1686.44140625 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2721.4045639038086 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 3.529296875# memory_percent‚92.0048713684082' proc.memory.availableMB‚ 1309.921875! disk.sda.out‚1731.93408203125 disk.nvme0n1.in‚ 7537.84375 disk.dm-0.in‚ 8055.95703125 disk.dm-0.out‚192894.9140625 disk.sdb2.in‚ 75.59765625 network.sent‚ 2.5350887e+07 disk.nvme1n1.in‚ 518.11328125! disk.nvme1n1.out‚ 53691.359375 disk.rbd2.in‚0 disk.md0.in‚ 208.765625( disk./.usagePercent‚76.0916106030467 disk.rbd0.in‚0! proc.memory.rssMB‚ 2977.140625ÆZ’Ž(‚(:‡ îÉëÔÐë¯à gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶$>Nv(‚(:ë îÉëÔ¹¨çà disk.sdb1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.06225357150659 disk.rbd2.out‚ 3.529296875" disk.nvme0n1.out‚ 139408.859375! disk.dm-0.out‚193100.46484375 cpu‚25.211299383169848 disk.sdb2.out‚ 1733.4609375 disk.sda.in‚ 133.16796875 disk.sda.out‚ 1733.4609375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1687.91796875 proc.cpu.threads‚17% disk./.usageGB‚2720.3546142578125 disk.sda1.in‚0 disk.md0.in‚ 208.765625 network.sent‚ 2.5355869e+07 disk.sdb.out‚ 1733.4609375 disk.md0p1.out‚ 1687.91796875 disk.sda2.in‚ 133.16796875 disk.sda2.out‚ 1733.4609375 network.recv‚ 3.805589e+06 disk.nvme0n1.in‚ 7537.84375$ memory_percent‚92.00491905212402! proc.memory.rssMB‚ 2977.140625 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚53691.60546875 disk.md0p1.in‚ 208.765625 disk.rbd2.in‚0 disk.dm-0.in‚ 8055.95703125( proc.memory.availableMB‚ 1309.9140625 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.out‚ 3.591796875C¢¸Ö”(‚(:‰ ýÉëÔ벋ß gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é ùæ_‘(‚(:Ô ýÉëÔÙãÿß disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17$ disk.nvme0n1.out‚139679.67578125 disk.md0p1.out‚ 1689.34375' proc.memory.availableMB‚ 1309.921875 disk.sdb.out‚ 1734.9375 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625 network.sent‚ 2.5359807e+07 disk.nvme1n1.out‚ 53691.78125 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 disk.nvme0n1.in‚ 7537.84375 disk.nvme1n1.in‚ 518.11328125 disk.sda2.out‚ 1734.9375 disk.dm-0.in‚ 8055.95703125 disk.sdb2.out‚ 1734.9375 disk.rbd0.out‚ 3.591796875 cpu‚25.211026754001303( disk./.usagePercent‚76.0622409855638$ disk./.usageGB‚2720.354164123535 disk.sdb2.in‚ 75.59765625 disk.sda1.out‚0! disk.dm-0.out‚193371.45703125 disk.sda.in‚ 133.16796875 disk.sda.out‚ 1734.9375 disk.sda1.in‚0 disk.rbd2.in‚0# memory_percent‚92.0048713684082 network.recv‚ 3.810135e+06 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1689.34375Dì³Ë4’(/ _wandbruntime‚11446  _runtime‚11446=×ÙÇ“(Á _runtime‚11446.303368943 _step‚781 & lr-AdamW/pg1‚9.413173855457975e-05 & lr-AdamW/pg2‚9.413173855457975e-05 trainer/global_step‚9549 $ _timestamp‚1.788536062767822e+09i–õEÌ”(Æ _step‚781 & lr-AdamW/pg1‚9.413173855457975e-05 & lr-AdamW/pg2‚9.413173855457975e-05 trainer/global_step‚9549 $ _timestamp‚1.788536062767822e+09 _runtime‚11446.303368943~<˜Ð4•(/ _wandbruntime‚11446  _runtime‚11446ÛèUq–(P `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776877791936528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241692643 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11411880974336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803415186412032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10929384421888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199363710 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241692643 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚776877774897168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887609365907 _runtime‚11446.304527878 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10888517340911 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199363708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199363710 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177230191 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441030007 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803415186412032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814343642859520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9549 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177230824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11412097173504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885360627688932e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241666299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897902340112896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚787765384263075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚886490483078656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚787766309277439 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚814344570833920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10929384421888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 441056353 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚814344570833920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803415168409088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10928474450432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814343642859520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10928474450432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803415168409088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57786182&ãê (—(†P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10928474450432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803415168409088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57786182 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776877791936528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241692643 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11411880974336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803415186412032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10929384421888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199363710 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241692643 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚776877774897168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887609365907 _runtime‚11446.304527878 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10888517340911 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199363708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199363710 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177230191 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441030007 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803415186412032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814343642859520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9549 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177230824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11412097173504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885360627688932e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241666299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897902340112896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚787765384263075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚886490483078656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚787766309277439 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚814344570833920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10929384421888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 441056353 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚814344570833920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803415168409088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10928474450432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814343642859520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Žö= ‹4˜(/ _wandbruntime‚11447  _runtime‚11447²¨9'™(‹N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803496805497856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897993751254016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199383870 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886580708783104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885360637665935e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177248025 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177248661 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814426383405568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241717500 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚776956792491712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 441101370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10929595910656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119456705 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441101370 trainer/global_step‚9549 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10930505895424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚787845519130897 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241691130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886580684843008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888726639185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897993991379968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11413282596864 _step‚783 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚803496787494912 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119456707 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803496805497856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897993751254016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199383870 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886580708783104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885360637665935e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177248025 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177248661 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚½Ò‡ 17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814426383405568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241717500 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚776956792491712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 441101370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10929595910656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119456705 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441101370 trainer/global_step‚9549 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10930505895424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚787845519130897 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241691130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886580684843008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888726639185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897993991379968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11413282596864 _step‚783 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚803496787494912 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119456707 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 Ex£(‚(:í ªÊëÔÄ­Àá$ memory_percent‚92.00682640075684 disk.md0.out‚ 1694.33984375 disk.dm-0.in‚ 8055.95703125 disk.md0p1.out‚ 1694.33984375 proc.cpu.threads‚17% disk./.usageGB‚2720.1492233276367 disk.rbd2.in‚0 disk.sda2.out‚ 1740.11328125 disk.sdb2.out‚ 1740.11328125 disk.sda.in‚ 133.16796875 disk.sda.out‚ 1740.11328125 disk.rbd2.out‚ 3.529296875# disk.nvme1n1.out‚53762.49609375 disk.sdb1.in‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.21034753048315) disk./.usagePercent‚76.05651075513609! disk.dm-0.out‚193597.51171875 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0* proc.memory.percent‚18.171024322509766" disk.nvme0n1.out‚ 139835.015625 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625 disk.sdb.out‚ 1740.11328125 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875( proc.memory.availableMB‚ 1309.6015625 disk.sda1.out‚0 network.sent‚ 2.547895e+07 network.recv‚ 3.825158e+06 disk.nvme0n1.in‚ 7537.84375 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sdb1.out‚0ô»˜™4¤(/ _wandbruntime‚11500  _runtime‚11500T$ñÇ¥(Á & lr-AdamW/pg1‚9.062333197105773e-05 & lr-AdamW/pg2‚9.062333197105773e-05 trainer/global_step‚9599 $ _timestamp‚1.788536117318039e+09 _runtime‚11500.853601374 _step‚785Á °Ì¦(Æ trainer/global_step‚9599 $ _timestamp‚1.788536117318039e+09 _runtime‚11500.853601374 _step‚785 & lr-AdamW/pg1‚9.062333197105773e-05 & lr-AdamW/pg2‚9.062333197105773e-05‘©ª4§(/ _wandbruntime‚11500  _runtime‚11500$âf(¨(P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443309255 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242937545 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178124586 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200371710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11470800234496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242911201 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807496140703232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10985471846912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚780828671663728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818481612550144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818481612550144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚791772141245383 trainer/global_step‚9599 _runtime‚11500.855534793 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10984561875456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120047892 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120047890 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807496140703232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚902472326299136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11470581938176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚780828654624368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807496122700288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200371708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178125230 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58077338 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10943486621015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242937545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818480684575744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818480684575744l)0~E©(†P trainer/global_step‚9599 _runtime‚11500.855534793 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10984561875456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120047892 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120047890 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807496140703232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚902472326299136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11470581938176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚780828654624368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807496122700288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200371708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178125230 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58077338 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10943486621015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242937545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818480684575744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818480684575744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443309255 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242937545 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178124586 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200371710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11470800234496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242911201 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807496140703232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated¿ö“Ç ‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10985471846912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚780828671663728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818481612550144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818481612550144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚791772141245383’Ñ{¥4ª(/ _wandbruntime‚11502  _runtime‚11502($g‰'«(‹N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818563425121792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10945511869505 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807577759789056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10985683335680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818564353109504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚807577759789056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443354272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 % _timestamp‚1.7885361184965708e+09 _runtime‚11502.033471565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443327900 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120059955 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10944603894293 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚9599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚891091994005504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10986593320448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178142444 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11471778606080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443327900 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚891091970065408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10985683335680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10986593320448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11471988500480¿­±é'¬(N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818564353109504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚807577759789056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443354272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 % _timestamp‚1.7885361184965708e+09 _runtime‚11502.033471565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443327900 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120059955 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10944603894293 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚9599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚891091994005504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10986593320448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178142444 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11471778606080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443327900 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚891091970065408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10985683335680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10986593320448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11471988500480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818563425121792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10945511869505 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807577759789056 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10985683335680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152“ë«.¾4­(/ _wandbruntime‚11502  _runtime‚11502³?6˜]®(× % _timestamp‚1.7885361184994543e+09 /loss/FRt2/assignment_loss‚3.332751091569662 _runtime‚11502.035037072 epoch‚7 /loss/FRt1/assignment_loss‚4.003646021797543 2loss/SRqqt2/assignment_loss‚1.2674384640924858 1loss/FBt2/assignment_loss‚0.26629503730988247 1loss/SRqqt3/detection_loss‚0.7901107887737453 /loss/FBt3/detection_loss‚0.5968325583120951 _step‚788 /loss/FBt1/detection_loss‚2.7238302528858185 1loss/SRqqt3/assignment_loss‚0.942641913125184 2loss/FBt4/assignment_loss‚0.029111984921172507 2loss/SRqqt1/assignment_loss‚1.1902706207973617 1loss/FBt1/assignment_loss‚0.36148461222018113 /loss/FRt1/detection_loss‚3.3431972492308843 0loss/FRt3/assignment_loss‚1.2685998678207397 0loss/FRt4/assignment_loss‚3.4448757476922944 1loss/SRqqt4/assignment_loss‚1.458095406663829 -loss/FRt2/detection_loss‚2.52223531551671 1loss/SRqqt4/detection_loss‚2.3760642039364783 /loss/FBt4/detection_loss‚2.0375804901123047 %loss/total_loss‚41.89860897246532 trainer/global_step‚9599 /loss/FRt3/detection_loss‚1.5765066297822876 1loss/SRqqt2/detection_loss‚2.3307110793662793 /loss/FBt2/detection_loss‚1.5591064426634047 1loss/FBt3/assignment_loss‚0.03279904716003028 -loss/FRt4/detection_loss‚2.21996201647491 1loss/SRqqt1/detection_loss‚2.2244621302400316Q¡ÞBb¯(Ü epoch‚7 /loss/FRt1/assignment_loss‚4.003646021797543 2loss/SRqqt2/assignment_loss‚1.2674384640924858 1loss/FBt2/assignment_loss‚0.26629503730988247 1loss/SRqqt3/detection_loss‚0.7901107887737453 /loss/FBt3/detection_loss‚0.5968325583120951 _step‚788 /loss/FBt1/detection_loss‚2.7238302528858185 1loss/SRqqt3/assignment_loss‚0.942641913125184 2loss/FBt4/assignment_loss‚0.029111984921172507 2loss/SRqqt1/assignment_loss‚1.1902706207973617 1loss/FBt1/assignment_loss‚0.36148461222018113 /loss/FRt1/detection_loss‚3.3431972492308843 0loss/FRt3/assignment_loss‚1.2685998678207397 0loss/FRt4/assignment_loss‚3.4448757476922944 1loss/SRqqt4/assignment_loss‚1.458095406663829 -loss/FRt2/detection_loss‚2.52223531551671 1loss/SRqqt4/detection_loss‚2.3760642039364783 /loss/FBt4/detection_loss‚2.0375804901123047 %loss/total_loss‚41.89860897246532 trainer/global_step‚9599 /loss/FRt3/detection_loss‚1.5765066297822876 1loss/SRqqt2/detection_loss‚2.3307110793662793 /loss/FBt2/detection_loss‚1.5591064426634047 1loss/FBt3/assignment_loss‚0.03279904716003028 -loss/FRt4/detection_loss‚2.21996201647491 1loss/SRqqt1/detection_loss‚2.2244621302400316 % _timestamp‚1.7885361184994543e+09 /loss/FRt2/assignment_loss‚3.332751091569662 _runtime‚11502.035037072”¼´É”°(‚(:‰ ¹ÊëÔþûÒß gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†'M…o±(‚(:ä ¹ÊëÔ›÷óß disk.md0p1.in‚ 208.765625) disk./.usagePercent‚76.05754173583958 disk.sdb.out‚ 1741.68359375 disk.md0p1.out‚ 1695.859375 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.210102410540568" disk.nvme0n1.out‚ 139835.015625 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8055.95703125 disk.rbd0.out‚ 3.591796875 network.sent‚ 2.5485226e+07 network.recv‚ 3.829858e+06 disk.md0.in‚ 208.765625 disk.dm-0.out‚ 193734.078125 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 1741.68359375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme1n1.out‚ 53899.0625 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚92.00544357299805' proc.memory.availableMB‚ 1309.828125 proc.cpu.threads‚17 disk.sda2.out‚ 1741.68359375 disk.md0.out‚ 1695.859375 disk.sdb2.out‚ 1741.68359375% disk./.usageGB‚2720.1860961914062 disk.nvme0n1.in‚ 7537.84375 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 518.11328125 disk.rbd2.out‚ 3.529296875* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 133.16796875Xp3“²(‚(:ˆ ÈÊëÔð³Üà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¥MQ³(‚(:ô ÈÊëÔúƒ—â disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.dm-0.out‚193907.5546875 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.sda1.out‚0 disk.md0.out‚ 1697.671875 disk.sdb2.in‚ 75.59765625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 network.recv‚ 3.83727e+06) disk./.usagePercent‚76.06716880880835 disk.rbd2.in‚0" disk.sdb2.out‚1743.54638671875! disk.sdb.out‚1743.54638671875 disk.nvme1n1.in‚ 518.11328125* proc.memory.percent‚18.171024322509766 disk.md0p1.out‚ 1697.671875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 3.529296875 disk.dm-0.in‚ 8055.95703125! disk.sda.out‚1743.54638671875) proc.memory.availableMB‚ 1309.94921875 proc.cpu.threads‚17 network.sent‚ 2.5595439e+07 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7537.84375# disk.nvme0n1.out‚139835.3984375 disk.nvme1n1.out‚ 54072.15625 disk.md0p1.in‚ 208.765625$ memory_percent‚92.00470447540283! proc.memory.rssMB‚ 2977.140625% disk./.usageGB‚2720.5304069519043" disk.sda2.out‚1743.54638671875 cpu‚25.209884016586326C¾¬Œ”´(‚(:‰ ×ÊëÔ›ÈÂà gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢õ!oµ(‚(:ä ×ÊëÔ²Êýà disk.dm-0.out‚ 194118.53125( disk./.usagePercent‚76.0776590854636 disk.sdb2.out‚ 1744.91796875 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1698.9765625* proc.memory.percent‚18.171024322509766 cpu‚25.20966034862579 proc.cpu.threads‚17 network.sent‚ 2.5598667e+07 network.recv‚ 3.840945e+06 disk.nvme0n1.in‚ 7537.84375 disk.rbd2.in‚0 disk.md0.out‚ 1698.9765625 disk.dm-0.in‚ 8055.95703125 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.sda1.in‚0 disk.rbd2.out‚ 3.529296875 disk.sdb.out‚ 1744.91796875# disk.nvme0n1.out‚139835.3984375" disk.nvme1n1.out‚ 54283.1328125! proc.memory.rssMB‚ 2977.140625 disk.sda2.out‚ 1744.91796875 disk.md0.in‚ 208.765625$ disk./.usageGB‚2720.905590057373 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 1744.91796875 disk.rbd0.in‚0$ memory_percent‚92.00401306152344% proc.memory.availableMB‚ 1310.0625\n”ž¶(‚(:“ æÊëÔöúÄà gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ápãüu·(‚(:ê æÊëÔŽ¡›â disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 133.16796875 disk.sdb.out‚ 1746.4296875 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625 cpu‚25.20944786389848 disk.nvme0n1.in‚ 7537.84375 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0* proc.memory.percent‚18.171024322509766 disk.sda1.out‚0 disk.sdb2.out‚ 1746.4296875$ disk./.usageGB‚2720.904628753662# disk.nvme0n1.out‚139835.6015625 disk.sda2.out‚ 1746.4296875 disk.sda.out‚ 1746.4296875 disk.nvme1n1.in‚ 518.11328125# disk.nvme1n1.out‚54538.37890625 disk.sdb1.out‚0! disk.dm-0.out‚194373.98046875 disk.rbd0.in‚0( proc.memory.availableMB‚ 1309.8203125) disk./.usagePercent‚76.07763220700949 disk.rbd2.in‚0 disk.md0.out‚ 1700.42578125 network.sent‚ 2.5606561e+07 network.recv‚ 3.844667e+06 disk.dm-0.in‚ 8055.95703125 disk.md0p1.out‚ 1700.42578125$ memory_percent‚92.00549125671387 proc.cpu.threads‚17ÕÊ=4¸(/ _wandbruntime‚11559  _runtime‚11559§3bFȹ( & lr-AdamW/pg2‚8.717229257784677e-05 trainer/global_step‚9649 % _timestamp‚1.7885361757209392e+09 _runtime‚11559.256437332 _step‚789 & lr-AdamW/pg1‚8.717229257784677e-05gE‡Gͺ(Ç & lr-AdamW/pg1‚8.717229257784677e-05 & lr-AdamW/pg2‚8.717229257784677e-05 trainer/global_step‚9649 % _timestamp‚1.7885361757209392e+09 _runtime‚11559.256437332 _step‚789•!Ø…4»(/ _wandbruntime‚11559  _runtime‚11559{·‰97¼(€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58368695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11529596204032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStat£]žÏ%sMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11559.257298636 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445562157 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179019837 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179019182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚895513029583360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120651142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795778898227691 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811577094994432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 $ _timestamp‚1.788536175721675e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244182447 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811577076991488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58368042 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445535811 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244182447 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784779534351568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907042401199616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811577076991488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11040649300480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784779551390928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244156103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11000271851127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822617726291968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907042649727488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11041559271936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811577094994432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201379708*1 (½(…P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚822618654266368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11041559271936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚822617726291968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445562157 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179019837 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179019182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚895513029583360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120651142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795778898227691 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811577094994432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 $ _timestamp‚1.788536175721675e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244182447 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811577076991488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58368042 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445535811 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244182447 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784779534351568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907042401199616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811577076991488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11040649300480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784779551390928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244156103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11000271851127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822617726291968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907042649727488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11041559271936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811577094994432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201379708 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58368695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11529596204032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11559.257298636 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179037061 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11001389124613 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244180934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚895603279227904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚907133795872768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907134046484480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445607174 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784858551946112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445607174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201399870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 179037714 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784858568985472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445580802 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚791 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811658714080256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11041770760704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811658714080256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11000481149401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201399868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445580802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201399868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244180934 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120663205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811658696077312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811658696077312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179037061 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11001389124613 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244180934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚895603279227904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚907133795872768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907134046484480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445607174 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784858551946112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445607174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201399870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 179037714 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784858568985472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445580802 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚791 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811658714080256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11041770760704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811658714080256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11000481149401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201399868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445580802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201399868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244180934 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120663205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 245185584 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11091373905408 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 121397175 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚816113100408832 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 245164873 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447766585 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚800117746233369 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11580525439488 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 % _timestamp‚1.7885362643302045e+09 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447787298 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 168404480 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447766585 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚800117048126561 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11090690866176 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11091373905408 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11049911013913 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚816113118411776 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 245185584 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚58757765 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚11580693843968 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚793 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚827204492317184 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚827203791275008 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 202601714 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚900345354861568 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121397173 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58757546 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚911925856360960 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚911926048705536 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚789067818180096 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚816113100408832 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 245185584 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11091373905408 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 121397175 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚816113100408832 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 245164873 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447766585 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚800117746233369 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11580525439488 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 % _timestamp‚1.7885362643302045e+09 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447787298 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 168404480 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447766585 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚800117048126561 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11090690866176 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11091373905408 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11049911013913 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚816113118411776 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 245185584 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚58757765 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚11580693843968 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚793 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚827204492317184 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚827203791275008 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 202601714 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚900345354861568 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121397173 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58757546 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚911925856360960 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚911926048705536 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚789067818180096 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615™ >8Œ4Ó(/ _wandbruntime‚11648  _runtime‚11648¬mBx)Ô(òR ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚827259243075072 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚11581183694848 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚827258542032896 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 168404480 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚816167379318784 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11091863756288 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚800167962442629 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚794 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚9671 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚900401598475264 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚789118261324640 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚58763088 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚11581015290368 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11050399224797 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447790875 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 121406735 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚911982806110208 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚800168660549437 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 202617188 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 447790875 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚816167379318784 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚900401622415360 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11049718157349 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11091863756288 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11091180717056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 202617188 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚827258542032896 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚789118244285280 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚11648.965162461 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 192344576 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 245173687 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 202617190 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180169604 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 245173687 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚58762869 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11091180717056 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 245194398 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚800167962442629 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚794 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚9671 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚900401598475264 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚789118261324640 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚58763088 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚11581015290368 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11050399224797 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447790875 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 121406735 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚911982806110208 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚800168660549437 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 202617188 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 447790875 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚816167379318784 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚900401622415360 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11049718157349 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11091863756288 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11091180717056 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 202617188 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚827258542032896 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚789118244285280 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚11648.965162461 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 192344576 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 245173687 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 202617190 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180169604 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 245173687 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚58762869 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11091180717056 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 245194398 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 particle/accuracy_3_of_5‚1 +particle/specificity‚0.9487391114234924 jet/accuracy_6_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7120313048362732 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4646868407726288 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4737262725830078 +jet/accuracy_2_of_3‚0.26849180459976196 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9874005317687988 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.045777175575494766 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5735088586807251 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FüÜùRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49983277916908264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2927771508693695 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023669345304369926 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,particle/sensitivity‚0.33781519532203674 particle/accuracy_5_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18824437260627747 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8611024618148804 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6178366541862488 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847849011421204 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 epoch‚7 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892041087150574 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38281217217445374 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4627681374549866 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4947497844696045 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07418651878833771 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013435201719403267 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5833333134651184 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2923734784126282 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13139016926288605 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08115926384925842 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1304597705602646 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33611106872558594 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03374727815389633 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 jet/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1261531263589859 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7500801682472229 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5393305420875549 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44131943583488464 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_4_of_4‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.034473419189453125 +jet/accuracy_3_of_6‚0.07692307978868484 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46429404616355896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4592677652835846 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4835766851902008 trainer/global_step‚9671 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44665902853012085 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5471248626708984 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18142718076705933 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017626726999878883 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5085011720657349 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881950616836548 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17095959186553955 particle/accuracy_2_of_3‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12815417349338531 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46429404616355896 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1623302549123764 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4595502018928528 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21273745596408844 *jet/accuracy_2_of_5‚0.4402521848678589 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.5512820482254028 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23456794023513794 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9738926887512207 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15532760322093964 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893191456794739 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.023190541192889214 particle/accuracy_1_of_3‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5387576222419739 *jet/accuracy_1_of_4‚0.9207979440689087 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02504105120897293 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4819895625114441 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3276591897010803 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5396435856819153 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5925925970077515 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10401163250207901 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021888384595513344 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40305474400520325 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443710327148438 :"Purity/1FRt2SRqqt1FBt/event_purity‚0.01149425283074379 +jet/accuracy_3_of_4‚0.08640865236520767 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_5‚0.951305627822876 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9755746722221375 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08860582113265991 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22204191982746124 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16599097847938538 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15181584656238556 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07459072023630142 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 jet/accuracy_2_of_7‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16921234130859375 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10065923631191254 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.47268953919410706 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20444446802139282 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4827300012111664 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17323729395866394 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2721264064311981 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12060439586639404 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.45319864153862 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916344285011292 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48017770051956177 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5458552241325378 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5455459952354431 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.46590909361839294 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5677397847175598 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5635676980018616 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48010697960853577 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5756579041481018 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2500743269920349 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19049982726573944 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5214151740074158 jet/accuracy_5_of_6‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚11665.653849282 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2308930605649948 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28578102588653564 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822256326675415 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986134767532349 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_2_of_5‚1 7validation_average_jet_accuracy‚0.43391016125679016 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4703310430049896 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 jet/accuracy_4_of_6‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24678993225097656 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08005952835083008 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11481264978647232 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5300387740135193 (particle/accuracy‚0.8533449769020081 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6294642686843872 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2580743730068207 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02047858014702797 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5115373730659485 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19322435557842255 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.079365074634552 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13677705824375153 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11520466953516006 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5090427994728088 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23852209746837616 'particle/f_score‚0.4181903004646301 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5408217906951904 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4757411777973175 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4965277910232544 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5950044393539429 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5056363344192505 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42305412888526917 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006504642777144909 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18024928867816925 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3837178945541382 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25009796023368835 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4956527054309845 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28457996249198914 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13631266355514526 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚795 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4581569731235504 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 particle/accuracy_4_of_6‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4681961238384247 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19933050870895386 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748501181602478 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4938271939754486 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0371926873922348 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37303048372268677 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17863403260707855 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5820969939231873 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4484673738479614 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33228760957717896 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897493124008179 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11011829227209091 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28703704476356506 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.009530792012810707 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚40.09164298321863 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5061327815055847 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5434020161628723 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45600607991218567 particle/accuracy_1_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9867283701896667 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1841578334569931 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13683126866817474 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24466994404792786 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37291663885116577 ,jet/accuracy_4_of_5‚0.005017921328544617 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4200892746448517 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43703705072402954 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4369395673274994 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5639471411705017 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03088061697781086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5623109936714172 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03006536141037941 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907535314559937 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969686269760132 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4951719343662262 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6121031641960144 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2417541891336441 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27037593722343445 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.506847083568573 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007988396100699902 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45185190439224243 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911478161811829 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22249415516853333 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5881943702697754 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1427203118801117 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17817038297653198 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4779660701751709 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49531012773513794 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_2_of_4‚0.39687058329582214 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4539687931537628 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033169545233249664 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5033164620399475 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9901477694511414 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19049982726573944 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20597080886363983 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1815776526927948 ;"Purit焵$®y/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5632184147834778 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4513888955116272 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3795061707496643 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 $ _timestamp‚1.788536282115764e+09 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024514993652701378 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9848405122756958 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5498705506324768 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46392109990119934 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37609684467315674 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3090342581272125 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0034486018121242523 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46962231397628784 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19691728055477142 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.205937460064888 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36187684535980225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5837301015853882 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12080184370279312 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4895833432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1RwhDqÚ(¡§ @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33611106872558594 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03374727815389633 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 jet/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1261531263589859 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7500801682472229 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5393305420875549 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44131943583488464 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_4_of_4‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.034473419189453125 +jet/accuracy_3_of_6‚0.07692307978868484 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46429404616355896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4592677652835846 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4835766851902008 trainer/global_step‚9671 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44665902853012085 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5471248626708984 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18142718076705933 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017626726999878883 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5085011720657349 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881950616836548 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17095959186553955 particle/accuracy_2_of_3‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12815417349338531 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46429404616355896 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1623302549123764 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4595502018928528 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21273745596408844 *jet/accuracy_2_of_5‚0.4402521848678589 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.5512820482254028 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23456794023513794 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9738926887512207 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15532760322093964 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893191456794739 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.023190541192889214 particle/accuracy_1_of_3‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5387576222419739 *jet/accuracy_1_of_4‚0.9207979440689087 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02504105120897293 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4819895625114441 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3276591897010803 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5396435856819153 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5925925970077515 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10401163250207901 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021888384595513344 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40305474400520325 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443710327148438 :"Purity/1FRt2SRqqt1FBt/event_purity‚0.01149425283074379 +jet/accuracy_3_of_4‚0.08640865236520767 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_5‚0.951305627822876 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9755746722221375 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08860582113265991 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22204191982746124 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16599097847938538 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15181584656238556 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07459072023630142 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 jet/accuracy_2_of_7‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16921234130859375 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10065923631191254 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.47268953919410706 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20444446802139282 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4827300012111664 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17323729395866394 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2721264064311981 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12060439586639404 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.45319864153862 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916344285011292 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48017770051956177 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5458552241325378 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5455459952354431 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.46590909361839294 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5677397847175598 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5635676980018616 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48010697960853577 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5756579041481018 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2500743269920349 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19049982726573944 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5214151740074158 jet/accuracy_5_of_6‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚11665.653849282 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2308930605649948 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28578102588653564 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822256326675415 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986134767532349 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_2_of_5‚1 7validation_average_jet_accuracy‚0.43391016125679016 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4703310430049896 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 jet/accuracy_4_of_6‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24678993225097656 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08005952835083008 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11481264978647232 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5300387740135193 (particle/accuracy‚0.8533449769020081 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6294642686843872 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2580743730068207 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02047858014702797 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5115373730659485 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19322435557842255 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.079365074634552 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13677705824375153 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11520466953516006 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5090427994728088 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23852209746837616 'particle/f_score‚0.4181903004646301 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5408217906951904 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4757411777973175 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4965277910232544 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5950044393539429 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5056363344192505 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42305412888526917 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006504642777144909 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18024928867816925 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3837178945541382 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25009796023368835 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4956527054309845 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28457996249198914 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13631266355514526 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚795 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4581569731235504 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 particle/accuracy_4_of_6‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4681961238384247 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19933050870895386 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748501181602478 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4938271939754486 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0371926873922348 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37303048372268677 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_6_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17863403260707855 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5820969939231873 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4484673738479614 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33228760957717896 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897493124008179 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11011829227209091 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28703704476356506 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.009530792012810707 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚40.09164298321863 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5061327815055847 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5434020161628723 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45600607991218567 particle/accuracy_1_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9867283701896667 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1841578334569931 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13683126866817474 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24466994404792786 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37291663885116577 ,jet/accuracy_4_of_5‚0.005017921328544617 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4200892746448517 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43703705072402954 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4369395673274994 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5639471411705017 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03088061697781086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5623109936714172 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03006536141037941 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907535314559937 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969686269760132 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4951719343662262 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6121031641960144 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2417541891336441 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27037593722343445 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.506847083568573 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007988396100699902 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45185190439224243 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911478161811829 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22249415516853333 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5881943702697754 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1427203118801117 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17817038297653198 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4779660701751709 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49531012773513794 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_2_of_4‚0.39687058329582214 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4539687931537628 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033169545233249664 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5033164620399475 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9901477694511414 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19049982726573944 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20597080886363983 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1815776526927948 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5632184147834778 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4513888955116272 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3795061707496643 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0  ld"particle/accuracy_3_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 $ _timestamp‚1.788536282115764e+09 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024514993652701378 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9848405122756958 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5498705506324768 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46392109990119934 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37609684467315674 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3090342581272125 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0034486018121242523 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46962231397628784 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19691728055477142 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.205937460064888 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36187684535980225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5837301015853882 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12080184370279312 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4895833432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 particle/accuracy_3_of_5‚1 +particle/specificity‚0.9487391114234924 jet/accuracy_6_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7120313048362732 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4646868407726288 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4737262725830078 +jet/accuracy_2_of_3‚0.26849180459976196 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9874005317687988 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.045777175575494766 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5735088586807251 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49983277916908264 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2927771508693695 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023669345304369926 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,particle/sensitivity‚0.33781519532203674 particle/accuracy_5_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18824437260627747 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8611024618148804 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6178366541862488 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847849011421204 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 epoch‚7 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892041087150574 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38281217217445374 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4627681374549866 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4947497844696045 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07418651878833771 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013435201719403267 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5833333134651184 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2923734784126282 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13139016926288605 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08115926384925842 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1304597705602646›œigû“Û(‚(:ˆ ÞËëÔýŠÄà gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æN“—‡Ü(‚(:ü ÞËëÔÚÊ¿á disk.md0p1.out‚ 1713.13671875 disk.sda1.in‚0 disk.sdb2.in‚ 75.59765625) proc.memory.availableMB‚ 1301.01953125 network.sent‚ 2.5854233e+07 network.recv‚ 3.883291e+06 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7537.84375$ disk./.usageGB‚2720.911029815674$ disk.nvme0n1.out‚140318.15234375 disk.md0.in‚ 208.765625 disk.dm-0.out‚ 196089.078125 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.11328125# proc.memory.rssMB‚ 2985.32421875 disk.sda1.out‚0 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1759.62646484375# disk.nvme1n1.out‚55770.92578125$ memory_percent‚92.05920696258545 proc.cpu.threads‚17( disk./.usagePercent‚76.0778111833824 disk.rbd2.in‚0" disk.sdb2.out‚1759.62646484375 cpu‚25.236141486670405 disk.md0p1.in‚ 208.765625* proc.memory.percent‚18.220973014831543 disk.rbd0.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1759.62646484375 disk.md0.out‚ 1713.13671875 disk.dm-0.in‚ 8055.95703125! disk.sda.out‚1759.62646484375r\±c’Ý(‚(:‡ íËëÔôÜÒà gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÏŒc|Þ(‚(:ñ íËëԇ߉á disk.md0p1.out‚ 1714.58203125# disk.nvme0n1.out‚140478.7890625 disk.sda2.in‚ 133.16796875 disk.dm-0.out‚196457.3046875! disk.nvme1n1.out‚ 55978.515625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.sda1.out‚0 disk.sda2.out‚1761.134765625 cpu‚25.23590485101938 network.sent‚ 2.5980211e+07 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 disk.sda.out‚1761.134765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚1761.134765625 disk.sdb.out‚1761.134765625 disk.rbd0.out‚ 3.591796875 disk.md0.out‚ 1714.58203125$ memory_percent‚92.05701351165771* proc.memory.percent‚18.220973014831543( disk./.usagePercent‚76.0778095834744 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.11328125) proc.memory.availableMB‚ 1301.37890625# proc.memory.rssMB‚ 2985.32421875 network.recv‚ 3.891012e+06$ disk./.usageGB‚2720.910972595215 disk.md0p1.in‚ 208.765625 disk.rbd2.out‚ 3.529296875 disk.nvme0n1.in‚ 7537.84375 disk.md0.in‚ 208.765625*B­n4ß(/ _wandbruntime‚11697  _runtime‚11697p”vÈà( & lr-AdamW/pg1‚8.377933956638376e-05 & lr-AdamW/pg2‚8.377933956638376e-05 trainer/global_step‚9699 % _timestamp‚1.7885363140295942e+09 _runtime‚11697.565271156 _step‚796WÂñmÍá(Ç & lr-AdamW/pg2‚8.377933956638376e-05 trainer/global_step‚9699 % _timestamp‚1.7885363140295942e+09 _runtime‚11697.565271156 _step‚796 & lr-AdamW/pg1‚8.377933956638376e-05œ§òÔã4â(/ _wandbruntime‚11697  _runtime‚11697J (ã(€P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819076468424192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180842346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245984563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚903569194634752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819076486427136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 449320915 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11128524290560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11086923499300 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11127614319104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11619433216512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121856796 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802994199244020 trainer/global_step‚9699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚915188368837632 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚830205010717696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚830205010717696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449347261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830204082743296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819076486427136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11128524290560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚830204082743296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245958219 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58985550 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788536314030795e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245984563 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245958219 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180842971 _step‚797 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚903569170694656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791907275744720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791907258705360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203362696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚11697.566801353 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203362698 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121856798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11086015524296 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203362698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203362696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58986173 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚915188627851264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830204082743296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819076486427136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11128524290560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚830204082743296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245958219 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58985550 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788536314030795e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245984563 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245958219 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180842971 _step‚797 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚903569170694656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791907275744720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791907258705360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203362696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚11697.566801353 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203362698 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121856798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11086015524296 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203362698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203362696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58986173 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚915188627851264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180842346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245984563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚903569194634752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819076486427136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 449320915 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11128524290560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11086923499300 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11127614319104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11619433216512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_,5 Eçstart/inactive_split.large_pool.freed‚ 121856796 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802994199244020 trainer/global_step‚9699"°U$4å(/ _wandbruntime‚11698  _runtime‚11698DýÇb'æ(‹N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11128735779328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819158087510016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11129645764096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180860098 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121868863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚903659420339200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203382856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58991862 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449365906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245983050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791986293339264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246009420 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚791986276299904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819158105512960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180860725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚915280015050240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915279758147072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11087132797574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449392278 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203382858 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚903659396399104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11620594711040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449392278 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11129645764096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245983050 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819158087510016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚830286823289344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11620361747968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203382856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚798 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚11698.711735382 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11128735779328 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180860098 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121868863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚903659420339200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203382856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58991862 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449365906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245983050 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791986293339264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 246009420 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚791986276299904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819158105512960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180860725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚915280015050240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915279758147072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11087132797574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449392278 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203382858 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚903659396399104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11620594711040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449392278 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11129645764096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245983050 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819158087510016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚830286823289344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11620361747968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203382856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚798 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚11698.711735382 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11128735779328 Füc”ñ(‚(:‰ ©ÌëÔæï³à gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷¸[ ‰ò(‚(:þ ©ÌëÔ¤Ôõá) proc.memory.availableMB‚ 1300.87109375 disk.sda1.in‚0 disk.dm-0.out‚197030.1328125" disk.sdb2.out‚1767.64111328125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625! disk.sdb.out‚1767.64111328125 disk.md0p1.out‚ 1720.84765625 disk.rbd2.out‚ 3.529296875$ disk.nvme0n1.out‚140807.96484375 disk.dm-0.in‚ 8055.95703125# disk.nvme1n1.out‚56222.16796875 disk.rbd0.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 2985.32421875) disk./.usagePercent‚76.05421328727418$ disk./.usageGB‚2720.067054748535 disk.nvme1n1.in‚ 518.11328125 disk.nvme0n1.in‚ 7537.84375 disk.sdb2.in‚ 75.59765625! disk.sda.out‚1767.64111328125 cpu‚25.234889636594954 proc.cpu.threads‚17 network.sent‚ 2.6105912e+07 network.recv‚ 3.910515e+06 disk.md0.out‚ 1720.84765625 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875$ memory_percent‚92.06011295318604* proc.memory.percent‚18.220973014831543 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0" disk.sda2.out‚1767.64111328125ÓÚ¦ð4ó(/ _wandbruntime‚11751  _runtime‚11751æDÑÇô(Á _runtime‚11751.774436595 _step‚800 & lr-AdamW/pg1‚8.044518002299107e-05 & lr-AdamW/pg2‚8.044518002299107e-05 trainer/global_step‚9749 $ _timestamp‚1.788536368238976e+09.¢»7Ìõ(Æ trainer/global_step‚9749 $ _timestamp‚1.788536368238976e+09 _runtime‚11751.774436595 _step‚800 & lr-AdamW/pg1‚8.044518002299107e-05 & lr-AdamW/pg2‚8.044518002299107e-05 Žë¤4ö(/ _wandbruntime‚11751  _runtime‚11751ŠXÎî ÷(P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204370696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11184611715584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚834341124459520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59271472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚795858155471920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451600163 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451573817 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834342052433920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚807000956226328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11142800754408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823157440718336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11677605451776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834341124459520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247229465 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚795858138432560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181732141 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11183701744128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204370698 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122460046 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11677368281088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7885363682400062e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204370698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247203121 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919758085308928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11184611715584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7885363682400062e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204370698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247203121 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919758085308928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11184611715584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451600163 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451573817 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834342052433920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚807000956226328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11142800754408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823157440718336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11677605451776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834341124459520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247229465 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚795858138432560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181732141 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11183701744128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204370698 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122460046 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¡0kÿº4ù(/ _wandbruntime‚11752  _runtime‚11752B‰“'ú(‹N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823239059804160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204390856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834424792993280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11185733189120 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122472111 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823239041801216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247254322 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451645180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795937156027104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11184823204352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204390858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204390858 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451645180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚919849475051520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807081091094358 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11143918027894 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181749812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11678769489920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834423865005568 trainer/global_step‚9749 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451618808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚834424792993280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚919849222342656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚908170705561600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11678540721152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚807080166079786 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247227952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181749189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11184823204352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908170681621504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204390856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181749189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11184823204352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908170681621504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204390856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823239059804160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204390856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834424792993280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11185733189120 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122472111 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823239041801216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247254322 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451645180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795937156027104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11184823204352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204390858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204390858 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451645180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚919849475051520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807081091094358 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11143918027894 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181749812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11678769489920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834423865005568 trainer/global_step‚9749 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144¢mZØî4ü(/ _wandbruntime‚11752  _runtime‚11752#*Ð cý(Ý 2loss/SRqqt4/assignment_loss‚1.4108522182214074 1loss/SRqqt1/detection_loss‚2.3252906143665313 /loss/FBt1/detection_loss‚2.4108412981033327 .loss/FBt2/detection_loss‚2.013129593928655 /loss/FRt3/detection_loss‚1.0990623301022673 1loss/FBt2/assignment_loss‚0.15220728098259617 0loss/FBt3/detection_loss‚0.35743536231284206 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚2.8272045368239995 0loss/FRt3/assignment_loss‚0.9766729245447132 1loss/FBt4/assignment_loss‚0.11560412152390806 0loss/SRqqt4/detection_loss‚2.234980635344982 %loss/total_loss‚37.86885284252869 1loss/SRqqt3/detection_loss‚0.9364678163922602 0loss/FRt1/assignment_loss‚2.1810300171375276 0loss/FRt4/assignment_loss‚3.0698083604083344 2loss/SRqqt3/assignment_loss‚0.9990957510026712 2loss/FBt3/assignment_loss‚0.029137965977644807 /loss/FRt2/detection_loss‚2.2498031979290922 /loss/FRt4/detection_loss‚2.6145061757415533 0loss/SRqqt2/detection_loss‚1.704090573248409 2loss/SRqqt1/assignment_loss‚1.2637282062942783 1loss/FBt1/assignment_loss‚0.13107521763540717 /loss/FBt4/detection_loss‚2.2833938241004943 epoch‚8 % _timestamp‚1.7885363692024243e+09 _runtime‚11752.737697589 2loss/SRqqt2/assignment_loss‚1.1491067978065639 /loss/FRt1/detection_loss‚3.3343280225992205 _step‚803qﺪhþ(â /loss/FBt1/detection_loss‚2.4108412981033327 .loss/FBt2/detection_loss‚2.013129593928655 /loss/FRt3/detection_loss‚1.0990623301022673 1loss/FBt2/assignment_loss‚0.15220728098259617 0loss/FBt3/detection_loss‚0.35743536231284206 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚2.8272045368239995 0loss/FRt3/assignment_loss‚0.9766729245447132 1loss/FBt4/assignment_loss‚0.11560412152390806 0loss/SRqqt4/detection_loss‚2.234980635344982 %loss/total_loss‚37.86885284252869 1loss/SRqqt3/detection_loss‚0.9364678163922602 0loss/FRt1/assignment_loss‚2.1810300171375276 0loss/FRt4/assignment_loss‚3.0698083604083344 2loss/SRqqt3/assignment_loss‚0.9990957510026712 2loss/FBt3/assignment_loss‚0.029137965977644807 /loss/FRt2/detection_loss‚2.2498031979290922 /loss/FRt4/detection_loss‚2.6145061757415533 0loss/SRqqt2/detection_loss‚1.704090573248409 2loss/SRqqt1/assignment_loss‚1.2637282062942783 1loss/FBt1/assignment_loss‚0.13107521763540717 /loss/FBt4/detection_loss‚2.2833938241004943 epoch‚8 % _timestamp‚1.7885363692024243e+09 _runtime‚11752.737697589 2loss/SRqqt2/assignment_loss‚1.1491067978065639 /loss/FRt1/detection_loss‚3.3343280225992205 _step‚803 2loss/SRqqt4/assignment_loss‚1.4108522182214074 1loss/SRqqt1/detection_loss‚2.3252906143665313£faþü”ÿ(‚(:‰ ¸ÌëÔ¹‘à gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÁÊíu€)‚(:ê ¸ÌëÔÎÜÅá disk.nvme1n1.in‚ 518.11328125 disk.sda1.in‚0 disk.sda2.out‚ 1769.38671875 disk.md0.in‚ 208.765625$ memory_percent‚92.06054210662842* proc.memory.percent‚18.220973014831543 disk.sdb1.out‚0 disk.sda.out‚ 1769.38671875 disk.rbd2.out‚ 3.529296875 disk.sdb2.out‚ 1769.38671875 network.recv‚ 3.914816e+06) proc.memory.availableMB‚ 1300.80078125% disk./.usageGB‚2720.0086708068848 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.591796875 disk.md0.out‚ 1722.53125 disk.sdb.in‚ 75.59765625 proc.cpu.threads‚17$ disk.nvme0n1.out‚140808.56640625 disk.sdb.out‚ 1769.38671875# proc.memory.rssMB‚ 2985.32421875 disk.dm-0.in‚ 8055.95703125 network.sent‚ 2.612274e+07 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 133.16796875) disk./.usagePercent‚76.05258084783006! disk.nvme1n1.out‚ 56284.546875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚197093.11328125 cpu‚25.23459224161828 disk.md0p1.out‚ 1722.53125 disk.sdb2.in‚ 75.59765625`¹‘)‚(:† ÇÌëÔ½Â„ß gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸)c9€‚)‚(:õ ÇÌëÔœäŽà disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚140808.56640625 disk.sda2.in‚ 133.16796875! disk.dm-0.out‚197231.27734375! disk.sdb2.out‚1771.3681640625 disk.md0p1.out‚ 1724.453125) disk./.usagePercent‚76.05988016134582 disk.sdb.in‚ 75.59765625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875# proc.memory.rssMB‚ 2985.32421875 proc.cpu.threads‚17 network.sent‚ 2.6217371e+07 network.recv‚ 3.921708e+06 disk.sdb1.in‚0 disk.sdb.out‚1771.3681640625 disk.rbd2.in‚0! disk.sda2.out‚1771.3681640625 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875" disk.nvme1n1.out‚ 56422.7109375# memory_percent‚92.0604944229126* proc.memory.percent‚18.220973014831543$ disk./.usageGB‚2720.269729614258 disk.rbd2.out‚ 3.529296875 disk.nvme0n1.in‚ 7537.84375 disk.sda.out‚1771.3681640625 disk.md0p1.in‚ 208.765625) proc.memory.availableMB‚ 1300.80859375 disk.md0.out‚ 1724.453125 disk.dm-0.in‚ 8055.95703125 disk.nvme1n1.in‚ 518.11328125 cpu‚25.234321627646654¬ ”Ú“ƒ)‚(:ˆ ÖÌëÔàìÙÞ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.666' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.9165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì’\q‡„)‚(:ü ÖÌëÔ„¨äß network.sent‚ 2.622298e+07) disk./.usagePercent‚76.06348710056058% disk./.usageGB‚2720.3987312316895 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1772.57080078125 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 7537.84375" disk.sda2.out‚1772.57080078125 disk.md0p1.out‚ 1725.59765625 disk.rbd2.in‚0$ disk.nvme0n1.out‚140808.63671875 disk.md0.out‚ 1725.59765625( proc.memory.availableMB‚ 1300.3046875* proc.memory.percent‚18.220973014831543! disk.sda.out‚1772.57080078125 disk.sdb1.in‚0 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1772.57080078125 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚197372.0546875# disk.nvme1n1.out‚56563.41796875 disk.md0p1.in‚ 208.765625$ memory_percent‚92.06357002258301 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 network.recv‚ 3.925399e+06 disk.dm-0.in‚ 8055.95703125 disk.nvme1n1.in‚ 518.11328125 disk.rbd2.out‚ 3.529296875 cpu‚25.23406792382233 disk.sda.in‚ 133.16796875# proc.memory.rssMB‚ 2985.32421875<±„¤”…)‚(:‰ åÌëÔ®©€ß gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì,i!}†)‚(:ò åÌëÔõªúà" disk.nvme0n1.out‚ 140808.640625! disk.sda2.out‚1774.0419921875$ memory_percent‚92.06054210662842 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625! disk.sdb2.out‚1774.0419921875 network.sent‚ 2.6231543e+07 disk.nvme1n1.in‚ 518.11328125 disk.rbd0.in‚0 disk.dm-0.out‚ 197534.96875# proc.memory.rssMB‚ 2985.32421875* proc.memory.percent‚18.220973014831543) disk./.usagePercent‚76.06698631263788 disk.sda.in‚ 133.16796875 disk.nvme0n1.in‚ 7537.84375 disk.sdb2.in‚ 75.59765625 disk.sdb.out‚1774.0419921875$ disk./.usageGB‚2720.523880004883 disk.sda.out‚1774.0419921875 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 network.recv‚ 3.930306e+06 disk.md0p1.out‚ 1727.0078125 disk.dm-0.in‚ 8055.95703125) proc.memory.availableMB‚ 1300.80078125 cpu‚25.233808194431255 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.out‚ 1727.0078125 disk.sdb.in‚ 75.59765625! disk.nvme1n1.out‚ 56726.328125 disk.rbd2.in‚0Î;¾×4‡)/ _wandbruntime‚11807  _runtime‚11807éxªoȈ) & lr-AdamW/pg1‚7.717050878151985e-05 & lr-AdamW/pg2‚7.717050878151985e-05 trainer/global_step‚9799 % _timestamp‚1.7885364235780902e+09 _runtime‚11807.113758582 _step‚804¿ ÃO͉)Ç _step‚804 & lr-AdamW/pg1‚7.717050878151985e-05 & lr-AdamW/pg2‚7.717050878151985e-05 trainer/global_step‚9799 % _timestamp‚1.7885364235780902e+09 _runtime‚11807.113758582¤N‰Ûd4Š)/ _wandbruntime‚11807  _runtime‚11807©·4(‹)€P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248474367 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 453853065 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚838479094150144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚838479094150144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11735681587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚838478166175744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123063298 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799809035199120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827238395009536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827238395009536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248474367 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912591741139456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838478166175744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827238377006592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11240699140608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924327422727168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11735906175488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11239789169152 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 182621668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚11807.11527653 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59558372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182622290 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123063296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205378698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚811007713208636 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9799 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123063298 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799809035199120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827238395009536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827238395009536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248474367 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912591741139456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838478166175744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827238377006592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11240699140608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924327422727168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11735906175488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11239789169152 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 182621668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚11807.11527653 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59558372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182622290 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123063296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.la·‰ö´s rge_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205378698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚811007713208636 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9799 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚838479094150144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¥ËËZ£4)/ _wandbruntime‚11808  _runtime‚11808¥º“'Ž)‹N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248499224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚11808.319261652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912681966843904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248472854 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799888035754304 _step‚806 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453898082 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11240910629376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123075363 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205398858 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827320014095360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182639382 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453898082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11737070756352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚811086923062094 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123075361 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11240910629376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827320014095360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11241820614144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827319996092416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11241820614144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 453871710 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11198887307790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205398856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205398858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚838561834709504 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182639995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924418804637184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453871710 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248472854 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11736837793280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚838560906721792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11240910629376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827320014095360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11241820614144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827319996092416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11241820614144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 453871710 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11198887307790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205398856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205398858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚838561834709504 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182639995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924418804637184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453871710 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248472854 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11736837793280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚838560906721792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182639382 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453898082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11737070756352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚811086923062094¦°xË4)/ _wandbruntime‚11808  _runtime‚11808Zû€^‘)Ø 1loss/FBt3/assignment_loss‚0.02889453578116142 0loss/SRqqt2/detection_loss‚2.091464569999112 .loss/FBt4/detection_loss‚2.168833553791046 /loss/FRt1/detection_loss‚2.8077538593395337 0loss/SRqqt1/detection_loss‚2.145529064110347 /loss/FBt1/detection_loss‚1.5828087215240185 2loss/SRqqt4/assignment_loss‚1.5634407275356352 epoch‚8 % _timestamp‚1.7885364247858942e+09 0loss/FRt1/assignment_loss‚2.5792734171893144 0loss/FRt3/assignment_loss‚0.7852905755862594 1loss/FBt1/assignment_loss‚0.22930123744338113 %loss/total_loss‚37.05117084422215 _step‚807 2loss/SRqqt1/assignment_loss‚1.4194671865020478 2loss/SRqqt2/assignment_loss‚1.3542374216227069 1loss/FBt2/assignment_loss‚0.21077108056414318 .loss/FRt2/detection_loss‚2.752169735590008 1loss/FBt4/assignment_loss‚0.08183370612221097 /loss/FRt3/detection_loss‚1.1106248773576226 1loss/SRqqt3/detection_loss‚0.6674234167066881 0loss/FRt2/assignment_loss‚2.8322657009829646 0loss/FRt4/assignment_loss‚2.8319283502046453 2loss/SRqqt3/assignment_loss‚0.9115840464206503 .loss/FRt4/detection_loss‚2.252702533938857 1loss/SRqqt4/detection_loss‚1.9555649044923484 /loss/FBt3/detection_loss‚0.4536210163931052 _runtime‚11808.321185171 .loss/FBt2/detection_loss‚2.234386605024338 trainer/global_step‚9799üÈÝpc’)Ý 0loss/SRqqt1/detection_loss‚2.145529064110347 /loss/FBt1/detection_loss‚1.5828087215240185 2loss/SRqqt4/assignment_loss‚1.5634407275356352 epoch‚8 % _timestamp‚1.7885364247858942e+09 0loss/FRt1/assignment_loss‚2.5792734171893144 0loss/FRt3/assignment_loss‚0.7852905755862594 1loss/FBt1/assignment_loss‚0.22930123744338113 %loss/total_loss‚37.05117084422215 _step‚807 2loss/SRqqt1/assignment_loss‚1.4194671865020478 2loss/SRqqt2/assignment_loss‚1.3542374216227069 1loss/FBt2/assignment_loss‚0.21077108056414318 .loss/FRt2/detection_loss‚2.752169735590008 1loss/FBt4/assignment_loss‚0.08183370612221097 /loss/FRt3/detection_loss‚1.1106248773576226 1loss/SRqqt3/detection_loss‚0.6674234167066881 0loss/FRt2/assignment_loss‚2.8322657009829646 0loss/FRt4/assignment_loss‚2.8319283502046453 2loss/SRqqt3/assignment_loss‚0.9115840464206503 .loss/FRt4/detection_loss‚2.252702533938857 1loss/SRqqt4/detection_loss‚1.9555649044923484 /loss/FBt3/detection_loss‚0.4536210163931052 _runtime‚11808.321185171 .loss/FBt2/detection_loss‚2.234386605024338 trainer/global_step‚9799 1loss/FBt3/assignment_loss‚0.02889453578116142 0loss/SRqqt2/detection_loss‚2.091464569999112 .loss/FBt4/detection_loss‚2.168833553791046 /loss/FRt1/detection_loss‚2.8077538593395337§VJÑh”“)‚(:‰ ôÌëÔÅàÇà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0énÑŽt”)‚(:é ôÌëÔÅ·Êâ disk.sda.in‚ 133.16796875! disk.nvme1n1.out‚ 56941.296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚140808.70703125 disk.sda2.in‚ 133.16796875 disk.sda2.out‚ 1775.76953125 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 1775.76953125 network.sent‚ 2.6235486e+07 disk.sda1.out‚0! disk.dm-0.out‚197750.00390625 disk.nvme1n1.in‚ 518.11328125 disk.rbd2.out‚ 3.529296875 disk.md0.in‚ 208.765625 disk.md0.out‚ 1728.65625 disk.sdb.in‚ 75.59765625 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1728.65625* proc.memory.percent‚18.220973014831543 cpu‚25.2335470132751$ disk./.usageGB‚2720.648670196533 disk.rbd2.in‚0 disk.sdb2.out‚ 1775.76953125 disk.rbd0.out‚ 3.591796875# proc.memory.rssMB‚ 2985.32421875 disk.nvme0n1.in‚ 7537.84375 disk.sdb.out‚ 1775.76953125 disk.rbd0.in‚0 network.recv‚ 3.934498e+06) disk./.usagePercent‚76.07047549862519$ memory_percent‚92.05930233001709) proc.memory.availableMB‚ 1301.00390625ŒVD¯’•)‚(:‡ ƒÍëÔäèà gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øâå«}–)‚(:ò ƒÍëÔü‚«á network.recv‚ 3.940931e+06 disk.dm-0.out‚ 197972.25 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1730.40625 disk.sdb1.out‚0" disk.sda2.out‚1777.58056640625' proc.memory.availableMB‚ 1301.046875 proc.cpu.threads‚17 disk.dm-0.in‚ 8055.95703125 disk.sda.in‚ 133.16796875 disk.sdb1.in‚0$ disk.nvme0n1.out‚140808.70703125 disk.sda2.in‚ 133.16796875* proc.memory.percent‚18.220973014831543 disk.nvme1n1.in‚ 518.11328125 cpu‚25.233299318116828 disk.md0.in‚ 208.765625 disk.rbd0.in‚0$ memory_percent‚92.05904006958008# proc.memory.rssMB‚ 2985.32421875) disk./.usagePercent‚76.06985612091516 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875% disk./.usageGB‚2720.6265182495117 disk.sdb.in‚ 75.59765625# disk.nvme1n1.out‚57163.54296875 disk.rbd0.out‚ 3.591796875 disk.md0.out‚ 1730.40625 disk.sdb2.in‚ 75.59765625 disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375! disk.sda.out‚1777.58056640625 network.sent‚ 2.6342837e+07! disk.sdb.out‚1777.58056640625" disk.sdb2.out‚1777.58056640625 disk.sda1.in‚0y¹‘”—)‚(:‰ ’ÍëԵŬà gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯VrU‚˜)‚(:÷ ’ÍëÔçºÑá disk.rbd2.out‚ 3.529296875 disk.sda.out‚1778.9423828125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1731.69921875 cpu‚25.233045379458417 disk.sda1.in‚0 disk.sdb.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1731.69921875 proc.cpu.threads‚17 network.sent‚ 2.6348554e+07 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚1778.9423828125 disk.sda.in‚ 133.16796875( proc.memory.availableMB‚ 1300.9296875) disk./.usagePercent‚76.06985078788856% disk./.usageGB‚2720.6263275146484 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 133.16796875 disk.dm-0.out‚ 198165.4375 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚57356.66015625$ disk.nvme0n1.out‚140808.77734375 disk.sdb.out‚1778.9423828125 disk.nvme1n1.in‚ 518.11328125$ memory_percent‚92.05975532531738 network.recv‚ 3.944891e+06! disk.sda2.out‚1778.9423828125 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8055.95703125# proc.memory.rssMB‚ 2985.32421875* proc.memory.percent‚18.220973014831543 disk.sdb1.out‚0ï §”™)‚(:‰ ¡ÍëÔ¼°£à gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´K8Ò†š)‚(:û ¡ÍëÔÐ÷à" disk.sdb2.out‚1780.43115234375" disk.nvme1n1.out‚ 57417.7734375 network.sent‚ 2.6354711e+07 disk.sda1.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8055.95703125 disk.md0p1.out‚ 1733.1171875) disk./.usagePercent‚76.08446157422071$ disk./.usageGB‚2721.148880004883 disk.rbd0.in‚0$ memory_percent‚92.05911159515381 disk.sdb2.in‚ 75.59765625! disk.sda.out‚1780.43115234375 disk.nvme1n1.in‚ 518.11328125* proc.memory.percent‚18.220973014831543 cpu‚25.232778004999858 disk.rbd2.in‚0$ disk.nvme0n1.out‚140808.86328125) proc.memory.availableMB‚ 1301.03515625 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0! disk.sdb.out‚1780.43115234375 disk.md0p1.in‚ 208.765625" disk.sda2.out‚1780.43115234375 disk.sda.in‚ 133.16796875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚ 3.529296875 disk.md0.out‚ 1733.1171875! disk.dm-0.out‚198226.63671875 disk.sdb.in‚ 75.59765625# proc.memory.rssMB‚ 2985.32421875 network.recv‚ 3.94962e+06 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7537.84375qæp•4›)/ _wandbruntime‚11865  _runtime‚11865Àgí¯Çœ)Á $ _timestamp‚1.788536481887548e+09 _runtime‚11865.423044974 _step‚808 & lr-AdamW/pg1‚7.395600827854883e-05 & lr-AdamW/pg2‚7.395600827854883e-05 trainer/global_step‚9849ŽM_LÌ)Æ & lr-AdamW/pg2‚7.395600827854883e-05 trainer/global_step‚9849 $ _timestamp‚1.788536481887548e+09 _runtime‚11865.423044974 _step‚808 & lr-AdamW/pg1‚7.395600827854883e-05¨,×WÁ4ž)/ _wandbruntime‚11865  _runtime‚11865rtŽÛoŸ)P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831319349300736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 249692925 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _LDeviceStÜ䶘%atsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11296786565632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803759897886960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11253647289620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885364818882573e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183512060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803759914926320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 456079621 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183512693 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123666546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206386698 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11296786565632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842615207891968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917103026361856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206386696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚917103050301952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚11865.423921448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 206386698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206386696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831319331297792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842616135866368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249719269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928897485267968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0"m/ ( )†P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249719269 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123666548 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803759914926320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 456079621 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183512693 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123666546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206386698 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11296786565632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842615207891968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917103026361856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206386696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚917103050301952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚11865.423921448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 206386698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206386696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831319331297792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842616135866368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249719269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928897485267968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831319349300736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 249692925 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11296786565632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803759897886960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11253647289620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885364818882573e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183512060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9849 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 456150984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831400968386560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456150984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚803838915481504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206406858 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚831400968386560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885364828614855e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123678613 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11297908039168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 456124612 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249717756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842698876425728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚928988609797632 _runtime‚11866.39764845 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123678611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11795596985856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249717756 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚842698876425728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206406856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚815094605058974 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59851263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183529874 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59851892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚917193252066304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803838932520864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚6154”Õrõ £)N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11296998054400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206406858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183530505 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831400968386560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456150984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚803838915481504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206406858 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚831400968386560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885364828614855e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123678613 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11297908039168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 456124612 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freedFVî ‚ 249717756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842698876425728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚928988609797632 _runtime‚11866.39764845 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123678611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11795596985856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249717756 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚842698876425728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206406856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚815094605058974 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59851263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183529874 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59851892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚917193252066304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803838932520864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 249744126 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚831400950383616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 456150984 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11351964019200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933467259853824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11310432519732 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚11923.531403863 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835400285588992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250937827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚846753177582592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250964171 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11352873990656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250964171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184404486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚921614335524352 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207394698 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835400285588992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60134067 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 % _timestamp‚1.7885365399953377e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207394696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846752249608192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207394698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚835400303591936 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124269796 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835400303591936 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11852924329472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11309524544728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 144011427843ª5Ã@²)†P `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚807710794653520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458332523 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11351964019200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933467259853824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11310432519732 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚11923.531403863 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835400285588992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250937827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚846753177582592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250964171 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11352873990656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250964171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184404486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚921614335524352 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207394698 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835400285588992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60134067 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 % _timestamp‚1.7885365399953377e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207394696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846752249608192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207394698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚835400303591936 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124269796 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835400303591936 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11852924329472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11309524544728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚846753177582592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458358869 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184403863 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚813 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60134688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9899 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_+‰W¬Ì batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚933558655959552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11353995464192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250989028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚814 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458403886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835481904674816 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207414856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250989028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846835918141952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚819101362041282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458377514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124281863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚846835918141952 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11311549793218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788536541124896e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11353995464192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207414858 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207414856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250962658 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60140388 trainer/global_step‚9899 _runtime‚11924.662647266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11353085479424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚933558405347840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921704561228800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835481922677760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835481904674816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚846834990154240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184421659 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250962658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184422251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921704537288704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207414858 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458403886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11353085479424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615ÿ×S'µ)N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚933558405347840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921704561228800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835481922677760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835481904674816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚846834990154240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184421659 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250962658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184422251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921704537288704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207414858 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458403886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11353085479424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11854094730752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚819100437026710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚933558655959552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11353995464192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250989028 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚814 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458403886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835481904674816 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207414856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250989028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846835918141952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚819101362041282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458377514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124281863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚846835918141952 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11311549793218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788536541124896e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11353995464192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207414858 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207414856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250962658 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60140388 trainer/global_step‚9899 _runtime‚11924.662647266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11353085479424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0®è2\G4¶)/ _wandbruntime‚11924  _runtime‚11924Œ“²ðb·)Ü epoch‚8 % _timestamp‚1.7885365411287408e+09 0loss/FRt3/assignment_loss‚1.3307492213589804 1loss/SRqqt1/detection_loss‚2.1969742959943312 /loss/FBt4/detection_loss‚2.6714411318302154 .loss/FRt4/detection_loss‚2.292889964999631 .loss/FRt1/detection_loss‚2.962893494853267 _runtime‚11924.664172223 2loss/FBt3/assignment_loss‚0.049318189498181814 1loss/SRqqt3/detection_loss‚0.7492771259344676 _step‚815 trainer/global_step‚9899 0loss/FRt1/assignment_loss‚2.6685205344800598 2loss/SRqqt1/assignment_loss‚1.1727693625244087 0loss/FBt2/assignment_loss‚0.3321120397390587 /loss/FRt3/detection_loss‚1.3617437221375959 1loss/SRqqt4/detection_loss‚2.2497514314723737 /loss/FBt1/detection_loss‚2.3326945419494924 /loss/FBt2/detection_loss‚2.0487198755145073 2loss/SRqqt3/assignment_loss‚0.9514208377485036 1loss/FBt1/assignment_loss‚0.09042317046802194 1loss/FBt4/assignment_loss‚0.17273381210115984 1loss/SRqqt2/detection_loss‚2.6262177423788953 &loss/total_loss‚39.825082453219366 0loss/FRt2/assignment_loss‚3.0112645044559385 2loss/SRqqt2/assignment_loss‚1.3686377019979632 /loss/FBt3/detection_loss‚0.4082608760282811 /loss/FRt4/assignment_loss‚3.106320550044378 2loss/SRqqt4/assignment_loss‚1.1615774677784154 /loss/FRt2/detection_loss‚2.5083708579312347Øés‚g¸)á 2loss/SRqqt4/assignment_loss‚1.1615774677784154 /loss/FRt2/detection_loss‚2.5083708579312347 epoch‚8 % _timestamp‚1.7885365411287408e+09 0loss/FRt3/assignment_loss‚1.3307492213589804 1loss/SRqqt1/detection_loss‚2.1969742959943312 /loss/FBt4/detection_loss‚2.6714411318302154 .loss/FRt4/detection_loss‚2.292889964999631 .loss/FRt1/detection_loss‚2.962893494853267 _runtime‚11924.664172223 2loss/FBt3/assignment_loss‚0.049318189498181814 1loss/SRqqt3/detection_loss‚0.7492771259344676 _step‚815 trainer/global_step‚9899 0loss/FRt1/assignment_loss‚2.6685205344800598 2loss/SRqqt1/assignment_loss‚1.1727693625244087 0loss/FBt2/assignment_loss‚0.3321120397390587 /loss/FRt3/detection_loss‚1.3617437221375959 1loss/SRqqt4/detection_loss‚2.2497514314723737 /loss/FBt1/detection_loss‚2.3326945419494924 /loss/FBt2/detection_loss‚2.0487198755145073 2loss/SRqqt3/assignment_loss‚0.9514208377485036 1loss/FBt1/assignment_loss‚0.09042317046802194 1loss/FBt4/assignment_loss‚0.17273381210115984 1loss/SRqqt2/detection_loss‚2.6262177423788953 &loss/total_loss‚39.825082453219366 0loss/FRt2/assignment_loss‚3.0112645044559385 2loss/SRqqt2/assignment_loss‚1.3686377019979632 /loss/FBt3/detection_loss‚0.4082608760282811 /loss/FRt4/assignment_loss‚3.106320550044378¯–ïsQ”¹)‚(:‰ ÝÍëÔ·ËŒâ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡î®„º)‚(:ù ÝÍëÔÂÊšã disk.sda1.out‚0! disk.sda.out‚1787.38134765625$ memory_percent‚92.06137657165527 cpu‚25.231806017156856# disk.nvme1n1.out‚58269.84765625# proc.memory.rssMB‚ 2985.32421875( disk./.usagePercent‚76.0914990361301 disk.nvme1n1.in‚ 518.1171875 disk.rbd0.out‚ 3.591796875 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 proc.cpu.threads‚17 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7537.84375 network.sent‚ 2.649221e+07% disk./.usageGB‚2721.4005737304688 disk.sdb1.out‚0 disk.rbd2.out‚ 3.529296875* proc.memory.percent‚18.220973014831543 disk.md0p1.out‚ 1739.80859375 disk.rbd0.in‚0 disk.sdb2.in‚ 75.59765625 network.recv‚ 3.968916e+06" disk.sda2.out‚1787.38134765625 disk.md0.out‚ 1739.80859375! disk.dm-0.out‚199079.23828125 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1787.38134765625" disk.sdb2.out‚1787.38134765625" disk.nvme0n1.out‚ 140809.390625 disk.dm-0.in‚ 8055.9609375 disk.sda.in‚ 133.16796875( proc.memory.availableMB‚ 1300.6640625é^玔»)‚(:‰ ìÍëÔƒ¢•à gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ž—Xx¼)‚(:í ìÍëÔ¸‚ªà! disk.sdb2.out‚1789.0166015625 disk.nvme1n1.in‚ 518.1171875 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 7537.84375 disk.nvme0n1.out‚ 140809.625) proc.memory.availableMB‚ 1300.37109375 disk.rbd0.in‚0* proc.memory.percent‚18.220973014831543 cpu‚25.23152786574702 network.sent‚ 2.6587486e+07 disk.dm-0.out‚ 199230.34375 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.sdb.out‚1789.0166015625 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.md0.out‚ 1741.38671875 disk.dm-0.in‚ 8055.9609375 disk.sdb.in‚ 75.59765625 disk.md0p1.out‚ 1741.38671875 disk.sdb1.in‚0$ memory_percent‚92.06316471099854! disk.sda2.out‚1789.0166015625 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.out‚ 58420.71875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2985.32421875 network.recv‚ 3.976581e+06$ disk./.usageGB‚2721.400680541992 disk.sda.out‚1789.0166015625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 3.529296875) disk./.usagePercent‚76.09150202262501^º’½)‚(:‡ ûÍëÔçí²à gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.352' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üújr¾)‚(:ç ûÍëÔÚæäá disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875 disk.sdb2.out‚ 1790.8203125 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 518.1171875 disk.md0p1.out‚ 1743.11328125* proc.memory.percent‚18.220973014831543 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7537.84375# proc.memory.rssMB‚ 2985.32421875( disk./.usagePercent‚76.0838841140998% disk./.usageGB‚2721.1282272338867 disk.md0.in‚ 208.765625 disk.sdb.out‚ 1790.8203125 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.out‚ 140809.625 disk.dm-0.in‚ 8055.9609375( proc.memory.availableMB‚ 1300.8671875 disk.rbd0.out‚ 3.591796875 disk.sda2.in‚ 133.16796875! disk.dm-0.out‚199475.21484375 disk.sdb.in‚ 75.59765625 cpu‚25.231294174971673 disk.sda.out‚ 1790.8203125# disk.nvme1n1.out‚58665.58984375 network.sent‚ 2.6595117e+07 disk.sda2.out‚ 1790.8203125 disk.md0.out‚ 1743.11328125$ memory_percent‚92.06013679504395 network.recv‚ 3.980312e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0Ì(´”¿)‚(:‰ ŠÎëÔø¸£à gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g)i^|À)‚(:ñ ŠÎëÔ´°Ùà disk.sda2.out‚ 1792.41796875 disk.dm-0.in‚ 8055.9609375 disk.sdb2.in‚ 75.59765625 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme1n1.out‚ 58821.7578125 disk.md0p1.in‚ 208.765625$ memory_percent‚92.05939769744873$ disk.nvme0n1.out‚140870.16796875 disk.md0p1.out‚ 1744.65234375 disk.sdb.out‚ 1792.41796875 disk.sda.in‚ 133.16796875) disk./.usagePercent‚76.08430595650435 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 7537.84375 disk.sdb.in‚ 75.59765625 network.recv‚ 3.984464e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875! disk.dm-0.out‚199691.92578125 disk.sda.out‚ 1792.41796875# proc.memory.rssMB‚ 2985.32421875 disk.rbd0.in‚0 disk.sdb2.out‚ 1792.41796875 network.sent‚ 2.6600792e+07 disk.sda2.in‚ 133.16796875 disk.nvme1n1.in‚ 518.1171875) proc.memory.availableMB‚ 1300.98828125* proc.memory.percent‚18.220973014831543 cpu‚25.231051352346178% disk./.usageGB‚2721.1433143615723 disk.md0.in‚ 208.765625 disk.md0.out‚ 1744.65234375““¥4Á)/ _wandbruntime‚11981  _runtime‚11981+ï—/ÈÂ) & lr-AdamW/pg2‚6.771018639735728e-05 trainer/global_step‚9949 % _timestamp‚1.7885365977979045e+09 _runtime‚11981.333416942 _step‚816 & lr-AdamW/pg1‚6.771018639735728e-05/ã³®ÍÃ)Ç _step‚816 & lr-AdamW/pg1‚6.771018639735728e-05 & lr-AdamW/pg2‚6.771018639735728e-05 trainer/global_step‚9949 % _timestamp‚1.7885365977979045e+09 _runtime‚11981.333416942°΢)ñ4Ä)/ _wandbruntime‚11981  _runtime‚11981a/JÅ)P QCDeviceStatsMonitor.CM^ßê'on_train_batch_start/active.small_pool.allocated‚ 252209073 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850890219298816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60424694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185298374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚926125596806656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460585425 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11408051444224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124873046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚839481257883136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885365977986689e+09 _runtime‚11981.334314696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850889291324416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208402698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚938036923943424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823027984155560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚811661674380720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850890219298816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60425326 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11911327136768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚817 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚926125620746752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208402696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11408051444224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252182729 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11408961415680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11365401799836 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 252209073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ç_ôb (Æ)†P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850889291324416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208402698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚938036923943424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823027984155560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚811661674380720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850890219298816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60425326 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11911327136768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚817 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚926125620746752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208402696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11408051444224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252182729 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11408961415680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11365401799836 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 252209073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252209073 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850890219298816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60424694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185298374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚926125596806656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460585425 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11408051444224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124873046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚839481257883136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885365977986689e+09 _runtime‚11981.334314696±cc –4Ç)/ _wandbruntime‚11982  _runtime‚11982EXé'È)‹N PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208422858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚926215846451200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11410082889216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚926215822511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460656788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185315607 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823107194009018 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208422858 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839562876968960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚938128314296320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850972959858176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460656788 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208422856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11410082889216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚839562858966016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252207560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460630416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252233930 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11366519073114 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124885113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208422856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60430496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839562858966016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124885111 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60431138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823108119023590 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850972031870464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11912491785216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640›×4é¢É)N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850972959858176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252233930 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208422858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚926215846451200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11410082889216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bªTstytes.large_pool.freed‚926215822511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460656788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185315607 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823107194009018 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208422858 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839562876968960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚938128314296320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850972959858176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460656788 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208422856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11410082889216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚839562858966016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252207560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460630416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252233930 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11366519073114 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124885113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208422856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60430496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839562858966016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124885111 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60431138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823108119023590 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850972031870464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11912491785216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855026333040640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462864673 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11970012590080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253453975 _runtime‚12034.838022456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125476296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570#·ªsÜÚ)…P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209410696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855026333040640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462864673 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11970012590080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253453975 _runtime‚12034.838022456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125476296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209410698 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843562212174336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚942607143147008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209410696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚843562194171392 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11421279054944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚942606894619136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788536651302442e+09µ@ö_°4Û)/ _wandbruntime‚12035  _runtime‚12035»ó}"'Ü)‹N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11466170314240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942698301262848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209430858 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11423304303434 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186208839 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125488363 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843643831260160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942698545583104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209430856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462883318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚822 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462909690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60721115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930727107733504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11971193529344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11465260329472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253478832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11465260329472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60720478 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚815691571702464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843643831260160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855110001574400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11466170314240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855110001574400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209430856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11422396328222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253452462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚9999 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186209478 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462909690 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚815691554663104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125488361 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930727131673600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827114876005898 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885366522622561e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855109073586688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971413909504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462883318 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚12035.798263555 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209430858˜¨Nó'Ý)N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125488361 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930727131673600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827114876005898 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885366522622561e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855109073586688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971413909504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462883318 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚12035.798263555 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209430858 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11466170314240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942698301262848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209430858 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11423304303434 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186208839 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125488363 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843643831260160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942698545583104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209430856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462883318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚822 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462909690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60721115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930727107733504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11971193529344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11465260329472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253478832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11465260329472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60720478 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚815691571702464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843643831260160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855110001574400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11466170314240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855110001574400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209430856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11422396328222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253452462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚9999 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186209478 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462909690 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚815691554663104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¶Çxä4Þ)/ _wandbruntime‚12035  _runtime‚12035§" >cß)Ý 0loss/FBt3/detection_loss‚0.41822174406115664 .loss/FRt1/detection_loss‚3.342628289352764 1loss/FBt1/assignment_loss‚0.15142922056270436 2loss/FBt3/assignment_loss‚0.014308431862446014 0loss/SRqqt1/detection_loss‚2.321358609380144 1loss/SRqqt2/detection_loss‚2.4075235770298886 0loss/SRqqt4/detection_loss‚2.437009693847762 2loss/SRqqt2/assignment_loss‚1.3450390057495007 /loss/FBt2/detection_loss‚2.6467765027826484 epoch‚8 _step‚823 2loss/SRqqt1/assignment_loss‚1.4486754865131595 /loss/FRt3/detection_loss‚1.4531312689347111 /loss/FRt1/assignment_loss‚3.824147018519315 0loss/FRt3/assignment_loss‚1.1266631872757622 1loss/FBt4/assignment_loss‚0.13283718288089782 1loss/SRqqt3/detection_loss‚0.7909219248309022 /loss/FBt4/detection_loss‚1.6844532907009124 trainer/global_step‚9999 _runtime‚12035.799850222 2loss/SRqqt3/assignment_loss‚1.0470970962994866 % _timestamp‚1.7885366522645717e+09 /loss/FRt2/detection_loss‚2.4432710805009785 /loss/FRt2/assignment_loss‚3.560622719355992 0loss/FRt4/assignment_loss‚2.8089735724709253 2loss/SRqqt4/assignment_loss‚1.2855871023817196 3loss/FBt2/assignment_loss‚0.0011381065234300795 .loss/FBt1/detection_loss‚3.524671159684658 &loss/total_loss‚42.534854365784255 /loss/FRt4/detection_loss‚2.3183690942823887â0©~hà)â /loss/FRt1/assignment_loss‚3.824147018519315 0loss/FRt3/assignment_loss‚1.1266631872757622 1loss/FBt4/assignment_loss‚0.13283718288089782 1loss/SRqqt3/detection_loss‚0.7909219248309022 /loss/FBt4/detection_loss‚1.6844532907009124 trainer/global_step‚9999 _runtime‚12035.799850222 2loss/SRqqt3/assignment_loss‚1.0470970962994866 % _timestamp‚1.7885366522645717e+09 /loss/FRt2/detection_loss‚2.4432710805009785 /loss/FRt2/assignment_loss‚3.560622719355992 0loss/FRt4/assignment_loss‚2.8089735724709253 2loss/SRqqt4/assignment_loss‚1.2855871023817196 3loss/FBt2/assignment_loss‚0.0011381065234300795 .loss/FBt1/detection_loss‚3.524671159684658 &loss/total_loss‚42.534854365784255 /loss/FRt4/detection_loss‚2.3183690942823887 0loss/FBt3/detection_loss‚0.41822174406115664 .loss/FRt1/detection_loss‚3.342628289352764 1loss/FBt1/assignment_loss‚0.15142922056270436 2loss/FBt3/assignment_loss‚0.014308431862446014 0loss/SRqqt1/detection_loss‚2.321358609380144 1loss/SRqqt2/detection_loss‚2.4075235770298886 0loss/SRqqt4/detection_loss‚2.437009693847762 2loss/SRqqt2/assignment_loss‚1.3450390057495007 /loss/FBt2/detection_loss‚2.6467765027826484 epoch‚8 _step‚823 2loss/SRqqt1/assignment_loss‚1.4486754865131595 /loss/FRt3/detection_loss‚1.4531312689347111·b·!ñ{á)‚(:ð ÕÎëÔÌâ$ disk./.usageGB‚2720.323356628418 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.70556640625 disk.sdb.out‚1800.365234375( proc.memory.availableMB‚ 1300.7578125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚1800.365234375 disk.sda.in‚ 133.16796875 disk.sda.out‚1800.365234375# proc.memory.rssMB‚ 2985.32421875 disk.rbd0.in‚0 disk.rbd0.out‚ 3.666015625 disk.dm-0.in‚ 8055.9609375 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 cpu‚25.22975483895957 disk.md0p1.out‚ 1752.30859375 disk.md0.in‚ 208.765625 network.sent‚ 2.6730222e+07 disk.nvme0n1.in‚ 7537.84375 disk.sda2.in‚ 133.16796875) disk./.usagePercent‚76.06137959510639 disk.nvme0n1.out‚ 141304.375 disk.md0.out‚ 1752.30859375! disk.dm-0.out‚200371.19140625$ memory_percent‚92.06080436706543* proc.memory.percent‚18.220973014831543 network.recv‚ 4.009091e+06 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sdb2.out‚1800.365234375 disk.nvme1n1.in‚ 518.1171875# disk.nvme1n1.out‚59066.81640625 proc.cpu.threads‚17Õócø“â)‚(:ˆ ÕÎëÔÓ¦á gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.386' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Àdö”ã)‚(:‰ äÎëÔѦ÷Þ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w·âŽä)‚(:ƒ  äÎëÔÁèöß) disk./.usagePercent‚76.05567464322479# disk.nvme1n1.out‚59067.61328125 disk.md0p1.out‚ 1754.16015625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7621.50390625 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1802.29052734375 disk.rbd2.in‚0 disk.md0.out‚ 1754.16015625 proc.cpu.threads‚17 disk.dm-0.out‚200523.4921875" disk.sdb2.out‚1802.29052734375 disk.sdb1.in‚0 disk.rbd2.out‚ 3.70556640625$ memory_percent‚92.06225872039795 cpu‚25.229520087559436" disk.sda2.out‚1802.29052734375 disk.dm-0.in‚ 8189.1484375 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875* proc.memory.percent‚18.220973014831543 network.recv‚ 4.01662e+06$ disk.nvme0n1.out‚141455.87890625 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2985.32421875 disk.rbd0.out‚ 3.7451171875 disk.nvme1n1.in‚ 567.64453125 network.sent‚ 2.6838469e+07% disk./.usageGB‚2720.1193199157715 disk.sda.in‚ 133.16796875! disk.sda.out‚1802.29052734375) proc.memory.availableMB‚ 1300.51953125ÿÍ‘å)‚(:† óÎëÔ¿ûºà gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]‡[æ)‚(:ÿ óÎëÔ›‘á disk.nvme1n1.in‚ 588.56640625* proc.memory.percent‚18.220973014831543 disk.md0p1.in‚ 208.765625 cpu‚25.229271557781207 disk.sda1.in‚0$ disk.nvme0n1.out‚141497.98828125 disk.sdb2.in‚ 75.59765625) disk./.usagePercent‚76.05636164371218 disk.rbd0.out‚ 3.7451171875 disk.sda1.out‚0$ memory_percent‚92.05808639526367' proc.memory.availableMB‚ 1301.203125 network.sent‚ 2.6843949e+07 disk.dm-0.in‚ 8231.64453125# disk.nvme1n1.out‚59128.27734375 proc.cpu.threads‚17 disk.dm-0.out‚ 200626.265625 disk.sdb.in‚ 75.59765625% disk./.usageGB‚2720.1438903808594 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 3.7451171875! disk.sdb.out‚1803.78271484375 disk.md0.in‚ 208.765625! disk.sda.out‚1803.78271484375 network.recv‚ 4.020444e+06 disk.md0p1.out‚ 1755.58984375 disk.sdb1.out‚0 disk.rbd2.in‚0" disk.sda2.out‚1803.78271484375# proc.memory.rssMB‚ 2985.32421875 disk.nvme0n1.in‚ 7643.078125 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1755.58984375" disk.sdb2.out‚æO/1803.78271484375 disk.sda.in‚ 133.16796875GÜý¸“ç)‚(:ˆ ‚ÏëÔ–Õðà gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–È‘»ˆè)‚(:ý ‚ÏëÔ„¡Ââ network.sent‚ 2.6849588e+07# disk.nvme0n1.out‚141550.1015625# proc.memory.rssMB‚ 2985.32421875 disk.sdb1.out‚0* proc.memory.percent‚18.220973014831543 disk.sdb2.in‚ 75.59765625 disk.sdb1.in‚0" disk.sda2.out‚1805.55224609375! disk.sda.out‚1805.55224609375 disk.nvme1n1.in‚ 594.11328125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.7451171875' proc.memory.availableMB‚ 1300.921875" disk.nvme1n1.out‚ 59152.5078125 disk.rbd0.out‚ 3.80810546875 disk.sda1.in‚0 cpu‚25.229022964035785) disk./.usagePercent‚76.06429558739406! disk.sdb.out‚1805.55224609375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7649.3203125 network.recv‚ 4.024281e+06$ disk./.usageGB‚2720.427646636963 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1757.29296875 disk.dm-0.out‚ 200702.609375 disk.sdb.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1757.29296875 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8243.43359375 disk.sda.in‚ 133.16796875# memory_percent‚92.0598030090332 proc.cpu.threads‚17" disk.sdb2.out‚1805.55224609375‚Ò'¹4é)/ _wandbruntime‚12091  _runtime‚12091ÙÃëÉê)à _step‚824 & lr-AdamW/pg1‚6.171292058783502e-05 & lr-AdamW/pg2‚6.171292058783502e-05 trainer/global_step‚10049 % _timestamp‚1.7885367079064717e+09 _runtime‚12091.442085675@²¤8Îë)È & lr-AdamW/pg1‚6.171292058783502e-05 & lr-AdamW/pg2‚6.171292058783502e-05 trainer/global_step‚10049 % _timestamp‚1.7885367079064717e+09 _runtime‚12091.442085675 _step‚824¸ßãí4ì)/ _wandbruntime‚12091  _runtime‚12091€}Õ(í)P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187085954 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11521136265728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187085321 _runtime‚12091.44320161 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚859164302731264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11478064285056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254672533 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12028702895104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚819563416795760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11477156310052 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465117575 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847643166465536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12028925385728 % _timestamp‚1.7885367079073112e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚847643148462592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61006406 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 465091229 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254672533 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210418696 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126079546 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254698877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210418698 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚947176870146560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚819563433835120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚859163374756864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11520226294272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847643148462592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11520226294272 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126079548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚859164302731264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚831040573105812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859163374756864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚819563416795760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11477156310052 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465117575 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847643166465536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12028925385728 % _timestamp‚1.7885367079073112e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚847643148462592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61006406 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 465091229 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254672533 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210418696 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126079546 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254698877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210418698 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚947176870146560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚819563433835120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚859163374756864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11520226294272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847643148462592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11520226294272 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126079548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚859164302731264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚831040573105812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859163374756864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚859164302731264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11478064285056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254672533¹r"%Š4ï)/ _wandbruntime‚12092  _runtime‚12092y˜ 'ð)ŒN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859246115302912 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126091611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819642434390304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859247043290624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847724767548416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210438856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚12092.439024282 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 465136220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12029879201792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11479181558542 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210438858 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210438856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11521347754496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚826 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847724785551360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187103769 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187103131 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465162592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254723734 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254697364 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11522257739264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254697364 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859247043290624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12030101679104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885367089028134e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847724767548416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚935238392955904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210438858 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210438858 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210438856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11521347754496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚826 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847724785551360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187103769 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187103131 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465162592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254723734 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254697364 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11522257739264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254697364 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859247043290624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12030101679104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885367089028134e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847724767548416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚935238392955904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210438858 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859246115302912 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126091611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819642434390304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859247043290624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847724767548416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210438856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚12092.439024282 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 465136220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12029879201792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11479181558542 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863300416473088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851724102753792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863301344447488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11577223690752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211426696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚851724102753792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255943779 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823514313562320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951747096096768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11533941540164 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187980469 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126682798 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12087403483648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12087619682816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11533033565160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255917435 _runtime‚12149.531665565 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211426698 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255917435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255943779 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187979842 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚10099 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126682796 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863300416473088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211426698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11576313719296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61297046 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835047330088120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823514296522960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640mwªXP€*‡P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187979842 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚10099 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126682796 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863300416473088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211426698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11576313719296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61297046 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835047330088120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823514296522960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11577223690752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 467344131 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11576313719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835048255102484 % _timestamp‚1.7885367659959059e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚851724120756736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863301344447488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863300416473088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851724102753792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863301344447488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11577223690752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211426696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚851724102753792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255943779 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823514313562320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951747096096768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11533941540164 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187980469 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126682798 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12087403483648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12087619682816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11533033565160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255917435 _runtime‚12149.531665565 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211426698 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255917435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255943779 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚12150.660935158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11577435179520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863383157019136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187997702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126694863 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835127464955942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851805721839616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467415494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61303467 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951838496970240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939749678178304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚851805739842560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851805721839616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚851805739842560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11577435179520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211446858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187998330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 trainer/global_step‚10099 % _timestamp‚1.7885367671237621e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939749702118400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12088557694464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823593331156864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11534150838438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467389122 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126694861 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚830 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255968636 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211446858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126694863 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835127464955942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851805721839616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467415494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61303467 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951838496970240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939749678178304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚851805739842560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851805721839616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚851805739842560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11577435179520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211446858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187998330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 trainer/global_step‚10099 % _timestamp‚1.7885367671237621e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939749702118400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12088557694464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823593331156864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11534150838438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467389122 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126694861 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚830 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255968636 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211446858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚12150.660935158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11577435179520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863383157019136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187997702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B˜È* & lr-AdamW/pg1‚5.596920987434434e-05 & lr-AdamW/pg2‚5.596920987434434e-05 trainer/global_step‚10149 % _timestamp‚1.7885368242127783e+09 _runtime‚12207.74841554 _step‚832A+›Í‘*Ç _runtime‚12207.74841554 _step‚832 & lr-AdamW/pg1‚5.596920987434434e-05 & lr-AdamW/pg2‚5.596920987434434e-05 trainer/global_step‚10149 % _timestamp‚1.7885368242127783e+09Àå4\k4’*/ _wandbruntime‚12207  _runtime‚12207_Òå(“*‚P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257188681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632401144320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632401144320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855805057044992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944170761636352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127286048 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127286046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867437458189312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚855805057044992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867437458189312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188874377 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956316844653568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839055012084792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867438386163712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 469623379 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚827465176250160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚827465193289520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212434698 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61587690 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212434696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11588910820268 trainer/global_step‚10149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188873736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615çœ (”*‡P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867437458189312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188874377 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956316844653568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839055012084792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867438386163712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 469623379 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚827465176250160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚827465193289520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212434698 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61587690 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212434696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11588910820268 trainer/global_step‚10149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188873736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257188681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632401144320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632401144320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855805057044992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944170761636352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127286048 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127286046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867437458189312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚855805057044992Á`vݬ4•*/ _wandbruntime‚12208  _runtime‚12208KLFæ·$–*ŒN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257213538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944260987340800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12147497963008 % _timestamp‚1.7885368253563638e+09 _runtime‚12208.893195417 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11633522604544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867521126723072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212454856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚956408215817728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11633522604544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469668396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61594116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469642024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 269486080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12147252417024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11590028093546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469642024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257213538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298111 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188891575 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188892229 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257187168 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212454858 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61593464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469668396 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855886694133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 245545984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11590936068758 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855886676130816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11634432589312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855886694133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11634432589312 >8DeviceSWFVˆ[tatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚839134221938250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127298113 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257187168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0a\ T'—*‘N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11634432589312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855886694133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11634432589312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚839134221938250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127298113 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257187168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257213538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944260987340800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12147497963008 % _timestamp‚1.7885368253563638e+09 _runtime‚12208.893195417 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11633522604544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867521126723072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212454856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚956408215817728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11633522604544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469668396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61594116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469642024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 269486080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12147252417024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11590028093546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469642024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257213538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298111 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188891575 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188892229 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257187168 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212454858 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61593464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469668396 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855886694133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 245545984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11590936068758 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855886676130816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Âàĸ4˜*/ _wandbruntime‚12208  _runtime‚12208ÂSã§a™*Û 2loss/SRqqt3/assignment_loss‚0.8719565026847778 1loss/SRqqt2/detection_loss‚2.2625752449035645 0loss/FRt1/assignment_loss‚3.4387903292973836 /loss/FRt3/assignment_loss‚1.184929934831766 0loss/FRt4/assignment_loss‚2.8578435340706183 /loss/FRt2/detection_loss‚2.5139136706225145 1loss/SRqqt4/detection_loss‚1.9059676271538402 .loss/FBt4/detection_loss‚2.348165382941564 2loss/SRqqt4/assignment_loss‚1.3490106938190238 .loss/FRt1/detection_loss‚2.782800793647766 /loss/FBt1/detection_loss‚2.0341777006785073 /loss/FBt3/detection_loss‚0.6821335854113731 2loss/SRqqt2/assignment_loss‚1.6072123238195977 1loss/FBt2/assignment_loss‚0.13780874829972162 2loss/FBt3/assignment_loss‚0.017796269132632738 1loss/SRqqt3/detection_loss‚0.7444890114017809 0loss/FRt2/assignment_loss‚2.5346733704209328 2loss/SRqqt1/assignment_loss‚1.5659070479182096 0loss/FBt1/assignment_loss‚0.2627047504139531 1loss/FBt4/assignment_loss‚0.06888351712322244 /loss/FRt3/detection_loss‚1.4119244048801751 _runtime‚12208.894496032 .loss/FRt4/detection_loss‚2.022470198875787 &loss/total_loss‚40.003311809012395 epoch‚8 _step‚835 1loss/SRqqt1/detection_loss‚2.3561400839915643 % _timestamp‚1.7885368253590667e+09 .loss/FBt2/detection_loss‚3.041037082672119 trainer/global_step‚10149U“Ħfš*à 2loss/SRqqt4/assignment_loss‚1.3490106938190238 .loss/FRt1/detection_loss‚2.782800793647766 /loss/FBt1/detection_loss‚2.0341777006785073 /loss/FBt3/detection_loss‚0.6821335854113731 2loss/SRqqt2/assignment_loss‚1.6072123238195977 1loss/FBt2/assignment_loss‚0.13780874829972162 2loss/FBt3/assignment_loss‚0.017796269132632738 1loss/SRqqt3/detection_loss‚0.7444890114017809 0loss/FRt2/assignment_loss‚2.5346733704209328 2loss/SRqqt1/assignment_loss‚1.5659070479182096 0loss/FBt1/assignment_loss‚0.2627047504139531 1loss/FBt4/assignment_loss‚0.06888351712322244 /loss/FRt3/detection_loss‚1.4119244048801751 _runtime‚12208.894496032 .loss/FRt4/detection_loss‚2.022470198875787 &loss/total_loss‚40.003311809012395 epoch‚8 _step‚835 1loss/SRqqt1/detection_loss‚2.3561400839915643 % _timestamp‚1.7885368253590667e+09 .loss/FBt2/detection_loss‚3.041037082672119 trainer/global_step‚10149 2loss/SRqqt3/assignment_loss‚0.8719565026847778 1loss/SRqqt2/detection_loss‚2.2625752449035645 0loss/FRt1/assignment_loss‚3.4387903292973836 /loss/FRt3/assignment_loss‚1.184929934831766 0loss/FRt4/assignment_loss‚2.8578435340706183 /loss/FRt2/detection_loss‚2.5139136706225145 1loss/SRqqt4/detection_loss‚1.9059676271538402 .loss/FBt4/detection_loss‚2.348165382941564ÿ´Ê”›*‚(:‰ úÏëÔôó²à gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßîOƒœ*‚(:ø úÏëÔ‡‚¼á network.sent‚ 2.7192451e+07 disk.md0p1.in‚ 208.765625" disk.sda2.out‚1818.79638671875 disk.sdb2.in‚ 75.59765625 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625! disk.sda.out‚1818.79638671875 network.recv‚ 4.064329e+06 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375# proc.memory.rssMB‚ 2985.32421875) disk./.usagePercent‚76.09138298947116 disk.nvme1n1.in‚ 850.171875 disk.sdb1.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚142587.97265625 disk.md0.out‚ 1770.03515625$ memory_percent‚92.06182956695557 cpu‚25.227107976926174 disk.sda1.in‚0 disk.dm-0.out‚202264.5# disk.nvme1n1.out‚59676.52734375 disk.md0p1.out‚ 1770.03515625 disk.rbd0.out‚ 3.87109375! disk.nvme0n1.in‚ 7673.98828125 disk.md0.in‚ 208.765625* proc.memory.percent‚18.220973014831543% disk./.usageGB‚2721.3964233398438 disk.dm-0.in‚ 8524.16015625 disk.sda.in‚ 133.16796875) proc.memory.availableMB‚ 1300.58984375 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.sdb.out‚1818.79638671875" disk.sdb2.out‚1818.79638671875è÷ÿ#“*‚(:ˆ ‰ÐëÔˆÇÌà gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.294' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.0735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 …½Þƒž*‚(:ø ‰ÐëÔʪ¡â disk.rbd2.out‚ 3.87109375 disk.sda1.out‚0% disk./.usageGB‚2721.3955841064453 disk.rbd0.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 1300.7578125! disk.sda.out‚1820.29150390625 network.sent‚ 2.729018e+07 network.recv‚ 4.071879e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 7674.2734375" disk.sda2.out‚1820.29150390625 disk.sdb2.in‚ 75.59765625# proc.memory.rssMB‚ 2985.32421875 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 8524.4453125 disk.dm-0.out‚ 202582.90625 disk.sdb.in‚ 75.59765625# disk.nvme1n1.out‚59787.62890625 proc.cpu.threads‚17) disk./.usagePercent‚76.09135952415409 disk.sdb1.out‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚142795.27734375$ memory_percent‚92.06080436706543* proc.memory.percent‚18.220973014831543 cpu‚25.226846886981754 disk.sda2.in‚ 133.16796875 disk.md0p1.out‚ 1771.46484375 disk.md0.out‚ 1771.46484375" disk.sdb2.out‚1820.29150390625 disk.md0p1.in‚ 208.765625 disk.md0.in‚ 208.765625! disk.sdb.out‚1820.29150390625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.171875Gi:÷‡Ÿ*‚(:ü ˜ÐëÔ×âÎà" disk.sda2.out‚1822.34912109375# proc.memory.rssMB‚ 2985.32421875 disk.rbd0.out‚ 3.87109375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚ 75.59765625" disk.nvme1n1.out‚ 59898.8359375 disk.rbd0.in‚0 disk.dm-0.out‚202797.8203125 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.18359375 cpu‚25.226576565385876 proc.cpu.threads‚17 network.sent‚ 2.7294363e+07! disk.sdb.out‚1822.34912109375 network.recv‚ 4.076356e+06 disk.sdb1.out‚0 disk.dm-0.in‚ 8524.45703125 disk.md0p1.out‚ 1773.45703125) proc.memory.availableMB‚ 1300.51953125" disk.nvme0n1.out‚ 142898.984375 disk.sdb1.in‚0" disk.sdb2.out‚1822.34912109375$ memory_percent‚92.06225872039795* proc.memory.percent‚18.220973014831543 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.md0.out‚ 1773.45703125 disk.md0p1.in‚ 208.765625 disk.nvme0n1.in‚ 7674.2734375) disk./.usagePercent‚76.09136112406208% disk./.usageGB‚2721.3956413269043 disk.rbd2.out‚ 3.87109375 disk.sda.in‚ 133.16796875! disk.sda.out‚1822.34912109375Ž d)” *‚(:‰ ˜ÐëÔÚ×à gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üZð+”¡*‚(:‰ §ÐëÔ‹â®à gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¨~’‰¢*‚(:þ §ÐëÔ¡ôƒá# proc.memory.rssMB‚ 2985.32421875! disk.dm-0.out‚203108.73046875 disk.sdb1.in‚0 disk.sdb2.in‚ 75.59765625 disk.rbd0.in‚0$ memory_percent‚92.06149578094482) proc.memory.availableMB‚ 1300.64453125* proc.memory.percent‚18.220973014831543 cpu‚25.226338621318767% disk./.usageGB‚2721.3955841064453 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8524.55078125 disk.rbd2.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.2890625 disk.sda1.in‚0 disk.rbd2.in‚0" disk.sdb2.out‚1823.86474609375 network.sent‚ 2.7299776e+07 network.recv‚ 4.080268e+06 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1823.86474609375! disk.nvme1n1.out‚ 60054.265625 disk.md0.out‚ 1774.91015625! disk.sda.out‚1823.86474609375 disk.rbd0.out‚ 3.87109375 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1823.86474609375 proc.cpu.threads‚17 disk.md0p1.out‚ 1774.91015625 disk.sdb1.out‚0$ disk.nvme0n1.out‚143054.46484375) disk./.usagePercent‚76.09135952415409 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.in‚ 208.765625u‹S4£*/ _wandbruntime‚12266  _runtime‚12266v=Úɤ*à % _timestamp‚1.7885368833785157e+09 _runtime‚12266.914864831 _step‚836 & lr-AdamW/pg1‚5.319394219277769e-05 & lr-AdamW/pg2‚5.319394219277769e-05 trainer/global_step‚10199ÛÒ­ÐÎ¥*È & lr-AdamW/pg1‚5.319394219277769e-05 & lr-AdamW/pg2‚5.319394219277769e-05 trainer/global_step‚10199 % _timestamp‚1.7885368833785157e+09 _runtime‚12266.914864831 _step‚836Ä(º/4¦*/ _wandbruntime‚12266  _runtime‚12266¸õ¢g(§*‚P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213442698 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚831416073016720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258407239 _step‚837 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11689398540800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11688488569344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843060844052736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚831416055977360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11645696050380 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61880696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843061769067100 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213442696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859886029339136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚871575427879936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471876281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚10199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚859886029339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859886011336192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960887095375360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213442696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885368833810747e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859886011336192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11689398540800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚960886832167424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12205048516608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12204809248768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258433583 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871574499905536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚948682022918656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859886011336192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960887095375360 D>Devic™+›­keStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213442696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885368833810747e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859886011336192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11689398540800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚960886832167424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12205048516608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12204809248768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258433583 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871574499905536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚948682022918656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11689398540800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11688488569344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843060844052736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚831416055977360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11645696050380 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61880696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843061769067100 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213442696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859886029339136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚871575427879936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471876281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚10199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚859886029339136 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189787216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871658168439296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871658168439296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚838 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 471921298 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚831495090611264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127901363 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859967648424960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚960978477162496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258458440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471921298 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213462856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189787847 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127901361 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11646813323866 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61886484 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258432070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948772272563200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843140978920558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471894926 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258432070 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859967630422016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11690520014336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚831495073571904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213462858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885368843767304e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚10199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213462856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859967630422016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948772272563200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843140978920558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471894926 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258432070 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859967630422016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11690520014336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚831495073571904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213462858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885368843767304e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚10199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213462856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859967630422016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189787216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871658168439296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871658168439296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚838 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 471921298 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚831495090611264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127901363 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859967648424960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚960978477162496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258458440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471921298 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213462856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189787847 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127901361 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11646813323866 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61886484Æšk¼G4¬*/ _wandbruntime‚12267  _runtime‚12267,ß„íY­*Ó % _timestamp‚1.7885368843793573e+09 1loss/SRqqt4/assignment_loss‚1.556523510851921 .loss/FRt2/detection_loss‚2.652440765165741 /loss/FRt3/detection_loss‚1.2335844806918543 /loss/FBt1/detection_loss‚2.4451675166686377 %loss/total_loss‚40.67932413365091 _step‚839 /loss/FRt3/assignment_loss‚1.194473281298598 1loss/SRqqt2/assignment_loss‚1.307387062021204 0loss/FBt1/assignment_loss‚0.1871926209710182 1loss/SRqqt4/detection_loss‚2.2774218152730894 .loss/FBt4/detection_loss‚2.553265407681465 trainer/global_step‚10199 epoch‚8 _runtime‚12267.914994095 0loss/FRt1/assignment_loss‚2.4323460973542312 /loss/FRt2/assignment_loss‚3.688319428970939 .loss/FRt1/detection_loss‚2.945052111970967 /loss/FRt4/detection_loss‚1.8796118395403028 1loss/SRqqt3/detection_loss‚0.7705987757523951 1loss/SRqqt1/assignment_loss‚1.387293137876051 /loss/SRqqt1/detection_loss‚2.41768791526556 0loss/FBt3/detection_loss‚0.47468801878505584 1loss/FBt2/assignment_loss‚0.15541842081656795 /loss/FRt4/assignment_loss‚3.447273831282343 1loss/SRqqt3/assignment_loss‚0.813871237582717 2loss/FBt3/assignment_loss‚0.039693489332090506 0loss/SRqqt2/detection_loss‚2.077826034378361 /loss/FBt2/detection_loss‚2.5278197050094606 1loss/FBt4/assignment_loss‚0.21436762911034393·^®*Ø 1loss/FBt2/assignment_loss‚0.15541842081656795 /loss/FRt4/assignment_loss‚3.447273831282343 1loss/SRqqt3/assignment_loss‚0.813871237582717 2loss/FBt3/assignment_loss‚0.039693489332090506 0loss/SRqqt2/detection_loss‚2.077826034378361 /loss/FBt2/detection_loss‚2.5278197050094606 1loss/FBt4/assignment_loss‚0.21436762911034393 % _timestamp‚1.7885368843793573e+09 1loss/SRqqt4/assignment_loss‚1.556523510851921 .loss/FRt2/detection_loss‚2.652440765165741 /loss/FRt3/detection_loss‚1.2335844806918543 /loss/FBt1/detection_loss‚2.4451675166686377 %loss/total_loss‚40.67932413365091 _step‚839 /loss/FRt3/assignment_loss‚1.194473281298598 1loss/SRqqt2/assignment_loss‚1.307387062021204 0loss/FBt1/assignment_loss‚0.1871926209710182 1loss/SRqqt4/detection_loss‚2.2774218152730894 .loss/FBt4/detection_loss‚2.553265407681465 trainer/global_step‚10199 epoch‚8 _runtime‚12267.914994095 0loss/FRt1/assignment_loss‚2.4323460973542312 /loss/FRt2/assignment_loss‚3.688319428970939 .loss/FRt1/detection_loss‚2.945052111970967 /loss/FRt4/detection_loss‚1.8796118395403028 1loss/SRqqt3/detection_loss‚0.7705987757523951 1loss/SRqqt1/assignment_loss‚1.387293137876051 /loss/SRqqt1/detection_loss‚2.41768791526556 0loss/FBt3/detection_loss‚0.47468801878505584Ç×MØ{¯*‚(:ð ¶ÐëÔÂ’Ùã disk.sdb.out‚1825.212890625" disk.nvme1n1.out‚ 60091.8984375 network.sent‚ 2.7318455e+07$ disk./.usageGB‚2721.123336791992 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875 disk.dm-0.out‚ 203282.484375) disk./.usagePercent‚76.08374737529763 disk.md0p1.out‚ 1776.19140625 disk.sdb.in‚ 75.59765625 disk.rbd0.out‚ 3.87109375# proc.memory.rssMB‚ 2985.32421875 proc.cpu.threads‚17 disk.sda2.out‚1825.212890625 disk.md0.in‚ 208.765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚143190.5859375 disk.md0.out‚ 1776.19140625 disk.sda.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625 disk.sdb2.out‚1825.212890625 disk.sda.out‚1825.212890625 disk.nvme0n1.in‚ 7674.2890625$ memory_percent‚92.06371307373047* proc.memory.percent‚18.220973014831543 network.recv‚ 4.084396e+06 disk.dm-0.in‚ 8524.55078125 disk.nvme1n1.in‚ 850.26171875 cpu‚25.226051050574437 disk.md0p1.in‚ 208.765625& proc.memory.availableMB‚ 1300.28125Yn¹”°*‚(:‰ ¶ÐëÔâÊŽå gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x‘M,ø±*‚(:‰ ÅÐëÔÕº¡à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0✠gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äfB|²*‚(:ö ÅÐëÔØÛÙá! disk.dm-0.out‚203458.16015625 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚60092.61328125 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875* proc.memory.percent‚18.220973014831543 network.recv‚ 4.091671e+06 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0" disk.nvme0n1.out‚ 143365.546875 disk.nvme1n1.in‚ 850.26171875! disk.sda2.out‚1826.8701171875 network.sent‚ 2.741368e+07 disk.md0p1.out‚ 1777.7734375 disk.nvme0n1.in‚ 7674.2890625 disk.md0.out‚ 1777.7734375 disk.sdb.out‚1826.8701171875 cpu‚25.225743530908588! disk.sdb2.out‚1826.8701171875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375 proc.cpu.threads‚17 disk.md0.in‚ 208.765625) disk./.usagePercent‚76.08349928289991 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0% disk./.usageGB‚2721.1144638061523 disk.sda.out‚1826.8701171875 disk.dm-0.in‚ 8524.55078125 disk.sdb.in‚ 75.59765625( proc.memory.availableMB‚ 1300.5234375 disk.sda1.in‚0$ memory_percent‚92.06223487854004# proc.memory.rssMB‚ 2985.32421875¨&é³*‚(:ö ÔÐëÔÒ•”ã disk.sdb.out‚1828.7158203125% disk./.usageGB‚2721.1768760681152 disk.md0p1.out‚ 1779.55078125 disk.rbd0.out‚ 3.87109375 disk.rbd0.in‚0# disk.nvme0n1.out‚143521.4140625 disk.sda2.in‚ 133.16796875 disk.sda.out‚1828.7158203125 disk.rbd2.out‚ 3.87109375 disk.dm-0.in‚ 8524.55078125! disk.sdb2.out‚1828.7158203125# proc.memory.rssMB‚ 2985.32421875* proc.memory.percent‚18.220973014831543 network.sent‚ 2.7419883e+07) disk./.usagePercent‚76.08524435586595 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625 cpu‚25.225483016300768 network.recv‚ 4.096086e+06 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.nvme1n1.out‚ 60153.28125 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7674.2890625 disk.md0.out‚ 1779.55078125 disk.sda1.out‚0! disk.sda2.out‚1828.7158203125( proc.memory.availableMB‚ 1300.6640625 proc.cpu.threads‚17 disk.sdb2.in‚ 75.59765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚203674.6953125$ memory_percent‚92.06137657165527D¢¼F”´*‚(:‰ ÔÐëÔçôŒã gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÝ]¡†µ*‚(:û ãÐëÔÄ’øà disk.rbd0.out‚ 3.87109375# disk.nvme0n1.out‚143521.4140625* proc.memory.percent‚18.220973014831543 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 network.sent‚ 2.742756e+07$ disk./.usageGB‚2721.395923614502 disk.sda1.in‚0 disk.md0.in‚ 208.765625! disk.dm-0.out‚203743.29296875 network.recv‚ 4.099875e+06" disk.sdb2.out‚1830.24755859375 disk.nvme1n1.in‚ 850.26171875 disk.rbd2.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.2890625 cpu‚25.22517766210293) disk./.usagePercent‚76.09136901694144! disk.sda.out‚1830.24755859375 disk.md0p1.out‚ 1781.01171875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 8524.55078125 disk.sda.in‚ 133.16796875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875$ memory_percent‚92.06137657165527 proc.cpu.threads‚17 disk.md0.out‚ 1781.01171875# disk.nvme1n1.out‚60221.87890625 disk.md0p1.in‚ 208.765625" disk.sda2.out‚1830.24755859375( proc.memory.availableMB‚ 1300.6640625# proc.memory.rssMB‚ 2985.32421875! disk.sdb.out‚1830.24755859375¾¿g“¶*‚(:ˆ ãÐëÔçžÛá gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.0705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þMŽ˜4·*/ _wandbruntime‚12325  _runtime‚12325KWŸMɸ*à _step‚840 & lr-AdamW/pg1‚5.048384193025137e-05 & lr-AdamW/pg2‚5.048384193025137e-05 trainer/global_step‚10249 % _timestamp‚1.7885369423037808e+09 _runtime‚12325.839262576µÔ‹Î¹*È & lr-AdamW/pg1‚5.048384193025137e-05 & lr-AdamW/pg2‚5.048384193025137e-05 trainer/global_step‚10249 % _timestamp‚1.7885369423037808e+09 _runtime‚12325.839262576 _step‚840È~ÿë+4º*/ _wandbruntime‚12325  _runtime‚12325Shj(»*‚P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 474102837 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 474129183 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259678485 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚847067601035044 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128492548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190663983 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚835366952743920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259652141 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚875712469596160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259652141 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 190664625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128492546 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11701573305488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259678485 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚863966965627392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚965456798260736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11744575994368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863966983630336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚965457048885760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚10249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚875711541621760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚12325.840452511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11700665330484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11744575994368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12263716804608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12263490119680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875711541621760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12263490119680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875711541621760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128492548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190663983 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚835366952743920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259652141 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚875712469596160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259652141 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 190664625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128492546 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11701573305488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259678485 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚863966965627392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚965456798260736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11744575994368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863966983630336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚965457048885760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚10249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚875711541621760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚12325.840452511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11700665330484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004Éü×Ò4½*/ _wandbruntime‚12326  _runtime‚12326³¾NÐ ¾*ŒN SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259703342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚875795210155520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚864048602716160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259676972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚10249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474147828 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12264639104000 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128504611 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚875795210155520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12264880455680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11702690578974 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190681722 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474174200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835445953299104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚847147735902866 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚835445970338464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11745697454592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 474147828 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚12326.788009243 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚864048584713216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62177111 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11746607439360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚965548438241280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953283557785600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190682365 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214470858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214470856 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259703342 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11745697454592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875794282167808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62177752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214470856T¯;ù'¿*‘N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474174200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835445953299104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚847147735902866 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚835445970338464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11745697454592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 474147828 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚12326.788009243 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚864048584713216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62177111 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11746607439360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚965548438241280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953283557785600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190682365 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214470858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214470856 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259703342 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11745697454592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875794282167808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62177752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214470856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259703342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚875795210155520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚864048602716160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259676972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚10249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474147828 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12264639104000 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128504611 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚875795210155520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12264880455680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11702690578974 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190681722 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Ê|K~4À*/ _wandbruntime‚12326  _runtime‚12326y§LZÁ*Ô 0loss/FBt3/detection_loss‚0.45253677293658257 2loss/SRqqt2/assignment_loss‚1.3252439766190947 /loss/FRt3/detection_loss‚1.3790867185841005 1loss/SRqqt3/detection_loss‚0.7376264456507503 % _timestamp‚1.7885369432542927e+09 _step‚843 /loss/FRt4/assignment_loss‚3.257730848648969 /loss/FRt2/detection_loss‚2.7995535984099433 /loss/FBt2/detection_loss‚2.1659204851497305 /loss/FBt4/detection_loss‚2.1529693139923944 0loss/FRt1/assignment_loss‚2.5707649205412184 1loss/SRqqt4/assignment_loss‚1.245573726017028 0loss/FRt2/assignment_loss‚3.0269654964407287 2loss/SRqqt1/assignment_loss‚1.1604498320817946 1loss/FBt3/assignment_loss‚0.01674291242777133 0loss/FBt4/assignment_loss‚0.2418228958914344 0loss/SRqqt4/detection_loss‚2.095255630241858 .loss/FBt1/detection_loss‚2.682740173556588 &loss/total_loss‚40.812481208813935 epoch‚8 0loss/FBt1/assignment_loss‚0.6477907134423565 .loss/SRqqt1/detection_loss‚2.3730197930336 _runtime‚12326.78958562 0loss/FBt2/assignment_loss‚0.5202742886980228 .loss/FRt4/detection_loss‚2.553526813602623 0loss/FRt3/assignment_loss‚1.2627040631529214 /loss/FRt1/detection_loss‚3.0984585732221603 1loss/SRqqt2/detection_loss‚2.0708218462765218 trainer/global_step‚10249 1loss/SRqqt3/assignment_loss‚0.974901370195748߬Úl_Â*Ù 0loss/FRt2/assignment_loss‚3.0269654964407287 2loss/SRqqt1/assignment_loss‚1.1604498320817946 1loss/FBt3/assignment_loss‚0.01674291242777133 0loss/FBt4/assignment_loss‚0.2418228958914344 0loss/SRqqt4/detection_loss‚2.095255630241858 .loss/FBt1/detection_loss‚2.682740173556588 &loss/total_loss‚40.812481208813935 epoch‚8 0loss/FBt1/assignment_loss‚0.6477907134423565 .loss/SRqqt1/detection_loss‚2.3730197930336 _runtime‚12326.78958562 0loss/FBt2/assignment_loss‚0.5202742886980228 .loss/FRt4/detection_loss‚2.553526813602623 0loss/FRt3/assignment_loss‚1.2627040631529214 /loss/FRt1/detection_loss‚3.0984585732221603 1loss/SRqqt2/detection_loss‚2.0708218462765218 trainer/global_step‚10249 1loss/SRqqt3/assignment_loss‚0.974901370195748 0loss/FBt3/detection_loss‚0.45253677293658257 2loss/SRqqt2/assignment_loss‚1.3252439766190947 /loss/FRt3/detection_loss‚1.3790867185841005 1loss/SRqqt3/detection_loss‚0.7376264456507503 % _timestamp‚1.7885369432542927e+09 _step‚843 /loss/FRt4/assignment_loss‚3.257730848648969 /loss/FRt2/detection_loss‚2.7995535984099433 /loss/FBt2/detection_loss‚2.1659204851497305 /loss/FBt4/detection_loss‚2.1529693139923944 0loss/FRt1/assignment_loss‚2.5707649205412184 1loss/SRqqt4/assignment_loss‚1.245573726017028Ë^¢‰ª’Ã*‚(:‡ òÐëÔ·‰á gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉpO†Ä*‚(:û òÐëÔ ‘×á disk.dm-0.in‚ 8524.55078125! disk.dm-0.out‚203988.42578125 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚60311.16015625 disk.md0p1.in‚ 208.765625$ disk./.usageGB‚2720.345832824707 disk.sdb.in‚ 75.59765625 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7431022e+07 network.recv‚ 4.103516e+06 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.2890625! disk.sdb.out‚1831.81494140625* proc.memory.percent‚18.220973014831543 disk.rbd2.in‚0 disk.md0.in‚ 208.765625! disk.sda.out‚1831.81494140625# proc.memory.rssMB‚ 2985.32421875) disk./.usagePercent‚76.06200803896164 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1300.15234375 disk.md0p1.out‚ 1782.51953125" disk.nvme0n1.out‚ 143677.265625 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1831.81494140625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.out‚ 3.87109375# memory_percent‚92.0644998550415 disk.md0.out‚ 1782.51953125" disk.sdb2.out‚1831.81494140625 cpu‚25.22492127598512Ø)é¡”Å*‚(:‰ ÑëÔÛŸòÞ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H(¼ˆÆ*‚(:ý ÑëÔ…´‡á$ disk.nvme0n1.out‚143764.45703125 disk.sdb.in‚ 75.59765625# disk.nvme1n1.out‚60400.08984375" disk.sdb2.out‚1833.82470703125 disk.sda.in‚ 133.16796875 network.recv‚ 4.111241e+06$ disk./.usageGB‚2720.345371246338 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.2890625 disk.md0.out‚ 1784.45703125 disk.dm-0.out‚ 204164.546875 network.sent‚ 2.7539127e+07 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.md0p1.out‚ 1784.45703125# proc.memory.rssMB‚ 2985.32421875 cpu‚25.224671389442907! disk.sdb.out‚1833.82470703125! disk.sda.out‚1833.82470703125 disk.md0p1.in‚ 208.765625* proc.memory.percent‚18.220973014831543 disk.sda1.in‚0 disk.md0.in‚ 208.765625) proc.memory.availableMB‚ 1300.66015625 proc.cpu.threads‚17) disk./.usagePercent‚76.06199513303726$ memory_percent‚92.06140041351318 disk.sdb1.in‚0 disk.sda1.out‚0" disk.sda2.out‚1833.82470703125 disk.sdb2.in‚ 75.59765625 disk.rbd2.out‚ 3.87109375 disk.dm-0.in‚ 8524.55078125 disk.nvme1n1.in‚ 850.26171875ì³×”Ç*‚(:‰ ÑëÔàù”á gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.317' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.57925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦‰"š{È*‚(:ð ÑëÔɪ®â disk.sdb.in‚ 75.59765625 disk.sdb.out‚ 1835.33984375 disk.sdb2.out‚ 1835.33984375# proc.memory.rssMB‚ 2985.32421875 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1785.92578125 disk.md0.in‚ 208.765625 cpu‚25.22441396724019 disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1300.29296875 disk.rbd2.in‚0# disk.nvme0n1.out‚143765.0078125 disk.dm-0.in‚ 8524.55078125! disk.dm-0.out‚204320.96484375 disk.sdb2.in‚ 75.59765625 proc.cpu.threads‚17$ disk./.usageGB‚2720.345375061035 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875$ memory_percent‚92.06364154815674( disk./.usagePercent‚76.0619952396978 disk.sda.out‚ 1835.33984375 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375* proc.memory.percent‚18.220973014831543 network.sent‚ 2.7547536e+07 disk.sda1.in‚0 disk.sda2.out‚ 1835.33984375 network.recv‚ 4.115751e+06# disk.nvme1n1.out‚60555.95703125 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.2890625 disk.md0.out‚ 1785.92578125‚Ñý”É*‚(:‰ ŸÑëÔ„úÞ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü‚Ê*‚(:÷ ŸÑëÔäôùâ disk.rbd2.in‚0 cpu‚25.224139891452257 network.sent‚ 2.7550865e+07 disk.sda2.in‚ 133.16796875 disk.nvme1n1.out‚ 60663.21875 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0$ memory_percent‚92.06211566925049* proc.memory.percent‚18.220973014831543 disk.md0.in‚ 208.765625) disk./.usagePercent‚76.05377619241355! disk.sdb2.out‚1836.7607421875 disk.nvme1n1.in‚ 850.26171875 network.recv‚ 4.119957e+06 disk.dm-0.in‚ 8524.55078125 disk.sdb.out‚1836.7607421875% disk./.usageGB‚2720.0514221191406 disk.dm-0.out‚204429.1796875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0! disk.sda2.out‚1836.7607421875 disk.md0.out‚ 1787.28515625 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1787.28515625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7674.2890625# disk.nvme0n1.out‚143765.9609375 disk.sdb.in‚ 75.59765625 disk.sda.out‚1836.7607421875 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375) proc.memory.availableMB‚ 1300.54296875# proc.memory.rssMB‚ 2985.32421875gø‡h4Ë*/ _wandbruntime‚12378  _runtime‚12378ŽF°ÆÌ*À _runtime‚12378.708267446 _step‚844 % lr-AdamW/pg1‚4.78394738675981e-05 % lr-AdamW/pg2‚4.78394738675981e-05 trainer/global_step‚10299 $ _timestamp‚1.788536995172731e+09ø½ËÍ*Å $ _timestamp‚1.788536995172731e+09 _runtime‚12378.708267446 _step‚844 % lr-AdamW/pg1‚4.78394738675981e-05 % lr-AdamW/pg2‚4.78394738675981e-05 trainer/global_step‚10299Ì:É <4Î*/ _wandbruntime‚12378  _runtime‚12378G |(Ï*€P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11800663419392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11800663419392 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215458696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚970026983307264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12322366003712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11756542585592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260897043 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879848583337984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851074358017352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚868047937921536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879849511312384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215458698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚957704593363456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191559244 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129095798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚12378.713035049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11801573390848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚868047919918592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 476382085 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62463446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62462821 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12322137221632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215458698 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260923387 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879848583337984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚868047937921536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11801573390848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260923387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚970026730585088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851075283031716 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11757450560596 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44ä–SÐ*…P VHDeviceStatsMonOó'itor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚12378.713035049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11801573390848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚868047919918592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 476382085 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62463446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62462821 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12322137221632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215458698 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260923387 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879848583337984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚868047937921536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11801573390848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260923387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚970026730585088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851075283031716 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11757450560596 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11800663419392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11800663419392 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215458696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚970026983307264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12322366003712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11756542585592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260897043 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879848583337984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851074358017352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚868047937921536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879849511312384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215458698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚957704593363456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191559244 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129095798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400Í.(à=4Ñ*/ _wandbruntime‚12379  _runtime‚12379€¨˜)'Ò*ŒN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879931323884032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476427102 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚868129539004416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11757659858870 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215478858 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879932251871744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚868129557007360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62468637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957794843008000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚10299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11758567834082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191576498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11802694864384 _step‚846 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879932251871744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚839396850065664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260948244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885369961609664e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191577129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12323320028672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215478856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 476400730 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260948244 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚839396833026304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0?ÝŒª'Ó*‘N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215478858 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879932251871744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚868129557007360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62468637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957794843008000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚10299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11758567834082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191576498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11802694864384 _step‚846 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879932251871744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚839396850065664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260948244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885369961609664e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191577129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12323320028672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215478856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 476400730 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260948244 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚839396833026304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879931323884032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476427102 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚868129539004416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11757659858870 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784ÎsÇi4Ô*/ _wandbruntime‚12379  _runtime‚12379uJSX_Õ*Ù 1loss/FBt4/assignment_loss‚0.10565282680421835 1loss/SRqqt3/detection_loss‚0.8210807796752099 .loss/FBt1/detection_loss‚3.128099908431371 /loss/FBt4/detection_loss‚2.1092158555984497 _step‚847 0loss/FRt2/assignment_loss‚2.7791550910031355 2loss/SRqqt4/assignment_loss‚1.0555596453034215 # _timestamp‚1.78853699616445e+09 0loss/FRt3/assignment_loss‚1.4669265658108157 trainer/global_step‚10299 2loss/SRqqt1/assignment_loss‚1.3012982991180921 2loss/SRqqt2/assignment_loss‚1.0845716720214114 /loss/FRt1/detection_loss‚2.7094359208897845 /loss/FRt4/detection_loss‚2.3158028748055752 1loss/SRqqt4/detection_loss‚1.9071025798718135 /loss/FBt2/detection_loss‚2.5347799241542814 $loss/total_loss‚39.6898829143948 /loss/FRt4/assignment_loss‚2.997906795956872 0loss/SRqqt2/detection_loss‚2.302223459817469 epoch‚8 0loss/FRt1/assignment_loss‚2.9342744466735096 1loss/FBt2/assignment_loss‚0.05407024877495133 2loss/FBt3/assignment_loss‚0.012480966298826272 0loss/SRqqt1/detection_loss‚2.794004473247026 1loss/FBt1/assignment_loss‚0.14956226909998804 /loss/FRt2/detection_loss‚2.3399373228420264 /loss/FRt3/detection_loss‚1.4267638042124349 _runtime‚12379.699770704 /loss/FBt3/detection_loss‚0.4163635148890788 2loss/SRqqt3/assignment_loss‚0.9436136690950445‰•\ìÖ*Þ /loss/FRt2/detection_loss‚2.3399373228420264 /loss/FRt3/detection_loss‚1.4267638042124349 _runtime‚12379.699770704 /loss/FBt3/detection_loss‚0.4163635148890788 2loss/SRqqt3/assignment_loss‚0.9436136690950445 1loss/FBt4/assignment_loss‚0.10565282680421835 1loss/SRqqt3/detection_loss‚0.8210807796752099 .loss/FBt1/detection_loss‚3.128099908431371 /loss/FBt4/detection_loss‚2.1092158555984497 _step‚847 0loss/FRt2/assignment_loss‚2.7791550910031355 2loss/SRqqt4/assignment_loss‚1.0555596453034215 # _timestamp‚1.78853699616445e+09 0loss/FRt3/assignment_loss‚1.4669265658108157 trainer/global_step‚10299 2loss/SRqqt1/assignment_loss‚1.3012982991180921 2loss/SRqqt2/assignment_loss‚1.0845716720214114 /loss/FRt1/detection_loss‚2.7094359208897845 /loss/FRt4/detection_loss‚2.3158028748055752 1loss/SRqqt4/detection_loss‚1.9071025798718135 /loss/FBt2/detection_loss‚2.5347799241542814 $loss/total_loss‚39.68988291ïRÈx43948 /loss/FRt4/assignment_loss‚2.997906795956872 0loss/SRqqt2/detection_loss‚2.302223459817469 epoch‚8 0loss/FRt1/assignment_loss‚2.9342744466735096 1loss/FBt2/assignment_loss‚0.05407024877495133 2loss/FBt3/assignment_loss‚0.012480966298826272 0loss/SRqqt1/detection_loss‚2.794004473247026 1loss/FBt1/assignment_loss‚0.14956226909998804χ^ôŽ”×*‚(:‰ ®ÑëÔÔÈ™á gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑòÜ»Ø*‚(:ö ®ÑëÔÓŠÝá( disk./.usagePercent‚76.0562693823519 disk.sdb1.out‚0 disk.md0.out‚ 1789.14453125 network.recv‚ 4.123693e+06 disk.sda1.out‚0# disk.nvme0n1.out‚143766.1796875 cpu‚25.22389443078813 disk.nvme0n1.in‚ 7674.2890625 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1838.68994140625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1789.14453125 disk.sdb.in‚ 75.59765625$ memory_percent‚92.06438064575195 network.sent‚ 2.756939e+07% disk./.usageGB‚2720.1405906677246 disk.sda.in‚ 133.16796875! disk.sda.out‚1838.68994140625 disk.sda1.in‚0# proc.memory.rssMB‚ 2985.32421875* proc.memory.percent‚18.220973014831543 disk.nvme1n1.in‚ 850.26171875" disk.nvme1n1.out‚ 60733.4765625 disk.rbd0.in‚0 disk.sda2.in‚ 133.16796875 proc.cpu.threads‚17" disk.sda2.out‚1838.68994140625 disk.dm-0.in‚ 8524.55078125! disk.sdb.out‚1838.68994140625' proc.memory.availableMB‚ 1300.171875 disk.dm-0.out‚ 204499.65625 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375ü 1’Ù*‚(:‡ ½ÑëÔõßÜß gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÏS1wÚ*‚(:ì ½ÑëÔ§œöà disk.sdb1.out‚0 disk.sda.in‚ 133.16796875# proc.memory.rssMB‚ 2985.32421875 disk.rbd2.in‚0# disk.nvme0n1.out‚143766.1796875 disk.sda2.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 1840.0390625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 7674.2890625 disk.md0.out‚ 1790.43359375$ disk./.usageGB‚2720.208869934082 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1790.43359375 disk.rbd0.out‚ 3.87109375 network.sent‚ 2.7664756e+07 disk.sda1.out‚0 disk.dm-0.in‚ 8524.55078125 disk.sdb2.out‚ 1840.0390625) proc.memory.availableMB‚ 1300.34765625* proc.memory.percent‚18.220973014831543 disk.sda2.out‚ 1840.0390625! disk.dm-0.out‚204588.66796875 disk.rbd0.in‚0) disk./.usagePercent‚76.05817849921634 disk.sdb.out‚ 1840.0390625# disk.nvme1n1.out‚60822.48828125" memory_percent‚92.063307762146 disk.md0.in‚ 208.765625 disk.md0p1.in‚ 208.765625 cpu‚25.223627601189783 network.recv‚ 4.130796e+06XÄ@Õ’Û*‚(:‡ ÌÑëÔÔöµà gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÅÃw€Ü*‚(:õ ÌÑëÔÐÈâ network.recv‚ 4.135245e+06 disk.dm-0.in‚ 8524.55078125 disk.dm-0.out‚ 204737.25 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1792.03515625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚1841.6943359375# disk.nvme1n1.out‚60970.86328125 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7674.2890625$ disk.nvme0n1.out‚143766.38671875) proc.memory.availableMB‚ 1299.98046875 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875$ memory_percent‚92.06554889678955# proc.memory.rssMB‚ 2985.32421875) disk./.usagePercent‚76.07043603422831 disk.md0.in‚ 208.765625 disk.sdb.out‚1841.6943359375 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sdb2.out‚1841.6943359375 disk.sda.in‚ 133.16796875 disk.rbd2.out‚ 3.87109375* proc.memory.percent‚18.220973014831543 cpu‚25.223353493531835 disk.md0.out‚ 1792.03515625 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 network.sent‚ 2.7673243e+07$ disk./.usageGB‚2720.647258758545! disk.sda2.out‚1841.6943359375 disk.md0p1.in‚ 208.765625K5ã“Ý*‚(:ˆ ÛÑëÔ´ýßá gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+íl#…Þ*‚(:ú ÛÑëԪǑä! disk.sdb.out‚1843.17529296875# proc.memory.rssMB‚ 2985.32421875* proc.memory.percent‚18.220973014831543 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 cpu‚25.223055425931115 proc.cpu.threads‚17 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 network.sent‚ 2.7678585e+07 disk.sdb2.in‚ 75.59765625 disk.md0p1.out‚ 1793.453125 disk.sdb1.out‚0 disk.rbd2.in‚0 network.recv‚ 4.139165e+06$ disk./.usageGB‚2720.897190093994 disk.sdb1.in‚0" disk.sda2.out‚1843.17529296875 disk.md0.out‚ 1793.453125) disk./.usagePercent‚76.07742421897186 disk.rbd0.out‚ 3.87109375 disk.dm-0.out‚204996.3515625# disk.nvme1n1.out‚61229.96484375 disk.rbd0.in‚0$ memory_percent‚92.06326007843018! disk.sda.out‚1843.17529296875 disk.md0p1.in‚ 208.765625 disk.rbd2.out‚ 3.87109375 disk.dm-0.in‚ 8524.56640625 disk.sdb.in‚ 75.59765625) proc.memory.availableMB‚ 1300.35546875" disk.sdb2.out‚1843.17529296875 disk.sda1.in‚0 disk.nvme0n1.in‚ 7674.3046875$ disk.nvme0n1.out‚143766.38671875 disk.md0.in‚ 208.765625}ãQ¨4ß*/ _wandbruntime‚12438  _runtime‚12438—‚§Éà*à _runtime‚12438.96209543 _step‚848 ' lr-AdamW/pg1‚4.5261389087158745e-05 ' lr-AdamW/pg2‚4.5261389087158745e-05 trainer/global_step‚10349 $ _timestamp‚1.788537055426422e+09º£ëÎá*È ' lr-AdamW/pg1‚4.5261389087158745e-05 ' lr-AdamW/pg2‚4.5261389087158745e-05 trainer/global_step‚10349 $ _timestamp‚1.788537055426422e+09 _runtime‚12438.96209543 _step‚848ÐðS(%4â*/ _wandbruntime‚12438  _runtime‚12438f\(ã*‚P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 262141945 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883986553028608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216466698 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚883985625054208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872128892212736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883985625054208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843268695158960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872128892212736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚974597018539008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚974596770011136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216466698 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192453543 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62755160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872128874209792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 478608641 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262168289 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843268712198320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478634987 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885370554277372e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883986553028608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚962215878585856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855081114999660Þ˜Mo (ä*‡P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885370554277372e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883986553028608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚962215878585856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855081114999660 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 262141945 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883986553028608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216466698 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚883985625054208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872128892212736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883985625054208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843268695158960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872128892212736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚974597018539008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚974596770011136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216466698 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192453543 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62755160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872128874209792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 478608641 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262168289 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843268712198320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478634987 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784Ñ ª²?4å*/ _wandbruntime‚12440  _runtime‚12440þ”æ*ŒN VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262193146 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12382041667072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚855161249867482 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216486858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62760874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216486856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192471987 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478680004 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192471332 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11858782289408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚872210511298560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884069293587968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚962306128230400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚10349 D6DeviceStatsMonitor.on_train_batch_end/active.all.fr\Ô¯‘eed‚ 478653632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚974688409152000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11858782289408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884068365600256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62760221 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚855162174882054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11857872304640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478680004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚962306104290304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262166776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843347712753504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11813537113978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚872210493295616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11857872304640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚12440.193444423 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974688145957376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216486856 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129711111 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129711113 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814445089190 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262166776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚850 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚855162174882054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11857872304640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478680004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚962306104290304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262166776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843347712753504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11813537113978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚872210493295616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11857872304640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚12440.193444423 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974688145957376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216486856 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129711111 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129711113 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814445089190 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262166776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚850 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62760874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216486856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192471987 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478680004 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192471332 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11858782289408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚872210511298560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884069293587968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚962306128230400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚10349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 478653632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚974688409152000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11858782289408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884068365600256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62760221Ò‰׌4è*/ _wandbruntime‚12440  _runtime‚12440ÿÀˆÚ]é*× /loss/FRt2/assignment_loss‚3.372192227479183 2loss/SRqqt2/assignment_loss‚1.5259901481680571 2loss/FBt3/assignment_loss‚0.061965197055159245 /loss/FRt1/assignment_loss‚3.133699434144156 0loss/FRt3/assignment_loss‚0.7938589970855152 1loss/SRqqt1/detection_loss‚2.3303690840458047 1loss/SRqqt4/detection_loss‚1.8983701653778553 0loss/FBt3/detection_loss‚0.46977579424549876 .loss/FBt4/detection_loss‚2.300221429930793 2loss/SRqqt3/assignment_loss‚0.9480758865922689 .loss/FRt1/detection_loss‚2.790997583951269 .loss/FRt4/detection_loss‚2.165979834198952 _runtime‚12440.201351647 2loss/SRqqt4/assignment_loss‚1.0030676952563227 0loss/FBt2/assignment_loss‚0.8416029036961845 1loss/SRqqt2/detection_loss‚1.8026511989533902 .loss/FBt1/detection_loss‚2.425535982847214 .loss/FBt2/detection_loss‚2.282204451660315 epoch‚8 0loss/FRt4/assignment_loss‚2.4504194343090058 2loss/SRqqt1/assignment_loss‚1.3326086469379992 /loss/FRt2/detection_loss‚2.5486286974804844 .loss/FRt3/detection_loss‚1.025740253574708 1loss/SRqqt3/detection_loss‚0.6573093383014202 _step‚851 1loss/FBt4/assignment_loss‚0.24542294396048683 $loss/total_loss‚38.5123692552699 1loss/FBt1/assignment_loss‚0.10568192601785995 % _timestamp‚1.7885370566593864e+09 trainer/global_step‚10349-¦ñbê*Ü .loss/FBt2/detection_loss‚2.282204451660315 epoch‚8 0loss/FRt4/assignment_loss‚2.4504194343090058 2loss/SRqqt1/assignment_loss‚1.3326086469379992 /loss/FRt2/detection_loss‚2.5486286974804844 .loss/FRt3/detection_loss‚1.025740253574708 1loss/SRqqt3/detection_loss‚0.6573093383014202 _step‚851 1loss/FBt4/assignment_loss‚0.24542294396048683 $loss/total_loss‚38.5123692552699 1loss/FBt1/assignment_loss‚0.10568192601785995 % _timestamp‚1.7885370566593864e+09 trainer/global_step‚10349 /loss/FRt2/assignment_loss‚3.372192227479183 2loss/SRqqt2/assignment_loss‚1.5259901481680571 2loss/FBt3/assignment_loss‚0.061965197055159245 /loss/FRt1/assignment_loss‚3.133699434144156 0loss/FRt3/assignment_loss‚0.7938589970855152 1loss/SRqqt1/detection_loss‚2.3303690840458047 1loss/SRqqt4/detection_loss‚1.8983701653778553 0loss/FBt3/detection_loss‚0.46977579424549876 .loss/FBt4/detection_loss‚2.300221429930793 2loss/SRqqt3/assignment_loss‚0.9480758865922689 .loss/FRt1/detection_loss‚2.790997583951269 .loss/FRt4/detection_loss‚2.165979834198952 _runtime‚12440.201351647 2loss/SRqqt4/assignment_loss‚1.0030676952563227 0loss/FBt2/assignment_loss‚0.8416029036961845 1loss/SRqqt2/detection_loss‚1.8026511989533902 .loss/FBt1/detection_loss‚2.425535982847214ÓÎR"k“ë*‚(:ˆ êÑëÔŒè·à gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜þsчì*‚(:ü êÑëÔü”ßá disk.sdb1.out‚0 disk.md0.in‚ 208.765625 network.recv‚ 4.142522e+06 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875! disk.sda.out‚1845.16259765625( proc.memory.availableMB‚ 1383.5859375 proc.cpu.threads‚17% disk./.usageGB‚2720.8962745666504 disk.md0p1.out‚ 1795.38671875$ memory_percent‚91.55526161193848! proc.memory.rssMB‚ 2902.359375# disk.nvme1n1.out‚61386.14453125 network.sent‚ 2.7693944e+07" disk.sda2.out‚1845.16259765625 disk.md0.out‚ 1795.38671875! disk.sdb.out‚1845.16259765625 cpu‚25.222797337513814$ disk.nvme0n1.out‚143766.62890625 disk.dm-0.out‚205152.7734375 disk.sdb2.in‚ 75.59765625* proc.memory.percent‚17.714595794677734 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 8524.56640625 disk.sdb.in‚ 75.59765625 disk.md0p1.in‚ 208.765625) disk./.usagePercent‚76.07739862044414 disk.sdb1.in‚0 disk.rbd2.in‚0" disk.sdb2.out‚1845.16259765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.3046875®°”í*‚(:‰ ùÑëÔèôß gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=î1û„î*‚(:ù ùÑëÔ£»×â# disk.nvme1n1.out‚61593.21484375 disk.md0p1.in‚ 208.765625 disk.dm-0.in‚ 8524.56640625 network.sent‚ 2.7791572e+07% disk./.usageGB‚2721.1261558532715 disk.sdb1.out‚0 disk.md0.in‚ 208.765625! proc.memory.rssMB‚ 2902.359375 disk.sda.in‚ 133.16796875! disk.dm-0.out‚205359.84765625# memory_percent‚91.5553331375122* proc.memory.percent‚17.714595794677734! disk.sda.out‚1846.58349609375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375) proc.memory.availableMB‚ 1383.57421875 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.3046875# disk.nvme0n1.out‚143766.6328125 proc.cpu.threads‚17 network.recv‚ 4.150482e+06) disk./.usagePercent‚76.08382619743085 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.md0p1.out‚ 1796.75390625 disk.md0.out‚ 1796.75390625 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1846.58349609375" disk.sda2.out‚1846.58349609375 cpu‚25.222525812284 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1846.58349609375Ð?Ö0‘ï*‚(:† ˆÒëÔéÝóà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[^Ýszð*‚(:ï ˆÒëÔ²‰åá disk.dm-0.out‚ 205553.546875 disk.md0p1.in‚ 208.765625# memory_percent‚91.5553331375122 network.recv‚ 4.154953e+06( disk./.usagePercent‚76.0826880228926# disk./.usageGB‚2721.08544921875 disk.rbd0.out‚ 3.87109375 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1847.990234375* proc.memory.percent‚17.714595794677734 disk.sda.out‚1847.990234375# disk.nvme1n1.out‚61786.74609375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.in‚ 208.765625$ disk.nvme0n1.out‚143766.80078125 disk.sda2.out‚1847.990234375 disk.sdb2.in‚ 75.59765625 disk.sdb2.out‚1847.990234375) proc.memory.availableMB‚ 1383.57421875! proc.memory.rssMB‚ 2902.359375 disk.md0.out‚ 1798.09765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1798.09765625 network.sent‚ 2.7795557e+07 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8524.56640625 cpu‚25.22226261079768 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7674.3046875§• ê”ñ*‚(:‰ —ÒëÔà•­á gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð½Á¥~ò*‚(:ó —ÒëÔ ‰—ä network.sent‚ 2.7803192e+07 disk.sda1.in‚0 disk.md0p1.out‚ 1800.16015625 cpu‚25.221986223434953# disk.nvme1n1.out‚61902.95703125 disk.sda1.out‚0 disk.nvme0n1.in‚ 7674.3046875 disk.md0.in‚ 208.765625 disk.sdb.out‚1850.138671875* proc.memory.percent‚17.714595794677734 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.nvme1n1.in‚ 850.26171875) disk./.usagePercent‚76.08438605856396 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 network.recv‚ 4.15879e+06% disk./.usageGB‚2721.1461791992188" disk.nvme0n1.out‚ 143827.265625 disk.md0.out‚ 1800.16015625 disk.dm-0.in‚ 8524.56640625 disk.md0p1.in‚ 208.765625 disk.sda2.in‚ 133.16796875 disk.sda2.out‚1850.138671875! disk.dm-0.out‚205730.22265625 disk.sda.out‚1850.138671875$ memory_percent‚91.55633449554443 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375 disk.sdb2.out‚1850.138671875! proc.memory.rssMB‚ 2902.359375 disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375) proc.memory.availableMB‚ 1383.41015625šÃ4ó*/ _wandbruntime‚12499  _runtime‚12499»øÿ¬Éô*à & lr-AdamW/pg1‚4.275012485793722e-05 & lr-AdamW/pg2‚4.275012485793722e-05 trainer/global_step‚10399 % _timestamp‚1.7885371157013316e+09 _runtime‚12499.236848197 _step‚852ŸY]Îõ*È & lr-AdamW/pg1‚4.275012485793722e-05 & lr-AdamW/pg2‚4.275012485793722e-05 trainer/global_step‚10399 % _timestamp‚1.7885371157013316e+09 _runtime‚12499.236848197 _step‚852ÔÇ$ä+4ö*/ _wandbruntime‚12499  _runtime‚12499Kù7à™÷*‚P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876209828500992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 480887889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚979167154668032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚12499.237775402 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63046773 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚859088796996332 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847219574886160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚888123594744832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚847219591925520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚10399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217474696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966727163808256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876209846503936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63046127 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217474696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚979166906140160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚888123594744832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130302296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193348423 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11869205070812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217474698 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚888122666770432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚859087871981968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11868297095808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876209828500992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876209846503936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11912838269440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11913748240896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 480861543 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11913748240896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12439742331904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚888122666770432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor. Íõoon_train_batch_start/active_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263413191 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11912838269440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚853 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263386847 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885371157021532e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 263386847 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193349071 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚966727187748352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 263413191 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚888123594744832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚847219591925520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚10399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217474696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966727163808256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876209846503936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63046127 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217474696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚979166906140160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚888123594744832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130302296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193348423 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11869205070812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217474698 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚888122666770432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚859087871981968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11868297095808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚876209828500992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876209846503936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11912838269440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11913748240896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 480861543 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11913748240896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12439742331904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚888122666770432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263413191 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11912838269440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚853 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263386847 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885371157021532e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 263386847 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193349071 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784ÕDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847298609520064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480932906 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263411678 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11913959729664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚854 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217494858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚888206335304192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚888206335304192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 263438048 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11914869714432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193366348 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217494858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63051987 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480932906 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847298592480704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11914869714432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480906534 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚888205407316480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217494856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12441141957632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚859168931864362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784e¯9P'û*‘N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚888206335304192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚888206335304192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 263438048 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11914869714432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193366348 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217494858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63051987 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480932906 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847298592480704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11914869714432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480906534 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚888205407316480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217494856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12441141957632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚859168931864362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚979258300604416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 trainer/global_step‚10399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859168006849790 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11870322344298 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847298609520064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480932906 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263411678 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11913959729664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚854 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217494858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615Ö[ F¥4ü*/ _wandbruntime‚12500  _runtime‚12500§yJ5Âý*Û 2loss/SRqqt1/assignment_loss‚1.6841879813901839 1loss/SRqqt3/detection_loss‚0.8166671215689608 0loss/FRt4/assignment_loss‚2.8109066112368715 2loss/FBt1/assignment_loss‚0.0©<þ"Ÿ18188288463685847 0loss/SRqqt4/detection_loss‚2.035500728806784 % _timestamp‚1.7885371168025131e+09 0loss/FRt1/assignment_loss‚2.6925502592517483 /loss/FRt1/detection_loss‚3.0863928621815098 .loss/FBt4/detection_loss‚2.171809023076838 _runtime‚12500.338096298 0loss/FRt3/assignment_loss‚1.4838155265897512 2loss/SRqqt4/assignment_loss‚1.4911949905718482 epoch‚8 2loss/FBt3/assignment_loss‚0.008253938628411974 1loss/FBt4/assignment_loss‚0.22345949927427516 -loss/FRt2/detection_loss‚2.53799746523071 2loss/SRqqt2/assignment_loss‚1.2304881570430903 2loss/SRqqt3/assignment_loss‚1.0029958212196979 0loss/FBt2/assignment_loss‚0.1492679210760709 /loss/FRt3/detection_loss‚1.3549765138304792 1loss/SRqqt2/detection_loss‚2.3575091786109486 /loss/FBt1/detection_loss‚3.0139944652716317 trainer/global_step‚10399 0loss/FRt2/assignment_loss‚3.4387487234213414 0loss/SRqqt1/detection_loss‚2.189166916954902 /loss/FBt2/detection_loss‚2.0876168229363183 %loss/total_loss‚40.55385424322475 /loss/FRt4/detection_loss‚2.1354464877615955 /loss/FBt3/detection_loss‚0.5327189388271008 _step‚855 µÆfþ*à 2loss/FBt3/assignment_loss‚0.008253938628411974 1loss/FBt4/assignment_loss‚0.22345949927427516 -loss/FRt2/detection_loss‚2.53799746523071 2loss/SRqqt2/assignment_loss‚1.2304881570430903 2loss/SRqqt3/assignment_loss‚1.0029958212196979 0loss/FBt2/assignment_loss‚0.1492679210760709 /loss/FRt3/detection_loss‚1.3549765138304792 1loss/SRqqt2/detection_loss‚2.3575091786109486 /loss/FBt1/detection_loss‚3.0139944652716317 trainer/global_step‚10399 0loss/FRt2/assignment_loss‚3.4387487234213414 0loss/SRqqt1/detection_loss‚2.189166916954902 /loss/FBt2/detection_loss‚2.0876168229363183 %loss/total_loss‚40.55385424322475 /loss/FRt4/detection_loss‚2.1354464877615955 /loss/FBt3/detection_loss‚0.5327189388271008 _step‚855 2loss/SRqqt1/assignment_loss‚1.6841879813901839 1loss/SRqqt3/detection_loss‚0.8166671215689608 0loss/FRt4/assignment_loss‚2.8109066112368715 2loss/FBt1/assignment_loss‚0.018188288463685847 0loss/SRqqt4/detection_loss‚2.035500728806784 % _timestamp‚1.7885371168025131e+09 0loss/FRt1/assignment_loss‚2.6925502592517483 /loss/FRt1/detection_loss‚3.0863928621815098 .loss/FBt4/detection_loss‚2.171809023076838 _runtime‚12500.338096298 0loss/FRt3/assignment_loss‚1.4838155265897512 2loss/SRqqt4/assignment_loss‚1.4911949905718482 epoch‚8×^‘äÔÿ*‚(:‰ ¦ÒëÔ–•á gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H2ñ~€+‚(:ó ¦ÒëÔôºËá network.recv‚ 4.162881e+06" disk.sdb2.out‚1851.56494140625 disk.rbd0.in‚0 disk.sda1.in‚0" disk.sda2.out‚1851.56494140625! disk.dm-0.out‚205890.64453125 disk.sdb.in‚ 75.59765625 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875* proc.memory.percent‚17.714595794677734 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0) proc.memory.availableMB‚ 1383.13671875 disk.sdb2.in‚ 75.59765625 disk.md0p1.out‚ 1801.53125 disk.sda1.out‚0! proc.memory.rssMB‚ 2902.359375 disk.rbd0.out‚ 3.87109375 disk.md0.in‚ 208.765625! disk.sdb.out‚1851.56494140625 proc.cpu.threads‚17 network.sent‚ 2.7806654e+07$ disk./.usageGB‚2721.271156311035# disk.nvme1n1.out‚61994.73828125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7674.3046875 disk.dm-0.in‚ 8524.56640625 cpu‚25.22173149249138 disk.sda.in‚ 133.16796875 disk.md0.out‚ 1801.53125) disk./.usagePercent‚76.08788047091734! disk.sda.out‚1851.56494140625 disk.nvme1n1.in‚ 850.26171875 disk.rbd2.in‚0! disk.nvme0n1.out‚ 143895.90625$ memory_percent‚91.55800342559814)bùe”+‚(:‰ µÒëÔ™„¨á gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.735' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.68375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ƒ‚+‚(:ø µÒëÔëä³ä disk.dm-0.in‚ 8524.56640625" disk.sdb2.out‚1853.13037109375 disk.nvme1n1.out‚ 62150.40625' proc.memory.availableMB‚ 1383.359375 cpu‚25.221434728537044 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7674.3046875 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.md0p1.out‚ 1803.04296875$ memory_percent‚91.55664443969727) disk./.usagePercent‚76.09137392332592$ disk.nvme0n1.out‚143984.62109375 disk.md0.in‚ 208.765625 disk.md0.out‚ 1803.04296875! disk.dm-0.out‚206135.02734375! disk.sda.out‚1853.13037109375 disk.md0p1.in‚ 208.765625 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375" disk.sda2.out‚1853.13037109375! disk.sdb.out‚1853.13037109375 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚ 3.87109375 network.sent‚ 2.7916986e+07 disk.rbd2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.714595794677734 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚2721.396099090576 network.recv‚ 4.170431e+06 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875! proc.memory.rssMB‚ 2902.359375o—Ø&”ƒ+‚(:‰ ÄÒëÔþùÝà gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s1#R}„+‚(:ò ÄÒëÔ¡ï–á$ disk./.usageGB‚2721.396156311035 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875! proc.memory.rssMB‚ 2902.359375 network.recv‚ 4.174988e+06 disk.sda.out‚1855.0107421875! disk.nvme0n1.out‚ 144073.71875 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625! disk.sdb2.out‚1855.0107421875 cpu‚25.221183854561122 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sdb.out‚1855.0107421875 disk.md0.out‚ 1804.86328125 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sda2.out‚1855.0107421875$ memory_percent‚91.55888557434082( proc.memory.availableMB‚ 1382.9921875 disk.sda.in‚ 133.16796875 disk.sda1.out‚0 disk.dm-0.in‚ 8524.56640625 proc.cpu.threads‚17 network.sent‚ 2.7920929e+07" disk.nvme1n1.out‚ 62358.3515625 disk.md0p1.out‚ 1804.86328125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 7674.3046875* proc.memory.percent‚17.714595794677734 disk.md0.in‚ 208.765625 disk.dm-0.out‚206432.0703125 disk.sdb1.out‚0( disk./.usagePercent‚76.0913755232339mš‹fž…+‚(:“ ÓÒëÔ±„Êà gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒhÏFu†+‚(:ê ÓÒëÔÃÅ‚â disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 8524.56640625 network.recv‚ 4.179254e+06 disk.sdb1.in‚0 disk.md0.out‚ 1806.20703125$ memory_percent‚91.55731201171875% disk./.usageGB‚2721.3961486816406 disk.sdb2.out‚ 1856.41015625 disk.sda.in‚ 133.16796875 disk.rbd2.out‚ 3.87109375 cpu‚25.22093650234466 disk.sdb.out‚ 1856.41015625# disk.nvme1n1.out‚62462.19921875 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.sda1.out‚0$ disk.nvme0n1.out‚144273.30859375* proc.memory.percent‚17.714595794677734 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7674.3046875 disk.dm-0.out‚206735.5078125! proc.memory.rssMB‚ 2902.359375 proc.cpu.threads‚17 network.sent‚ 2.7926586e+07 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1806.20703125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚ 1856.41015625) disk./.usagePercent‚76.09137530991285 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 1856.41015625# proc.memory.availableMB‚1383.25«±Áß4‡+/ _wandbruntime‚12559  _runtime‚12559„Ç-Ȉ+ & lr-AdamW/pg1‚4.030620452363502e-05 & lr-AdamW/pg2‚4.030620452363502e-05 trainer/global_step‚10449 $ _timestamp‚1.788537176108394e+09 _runtime‚12559.643949537 _step‚856´BUï͉+Ç & lr-AdamW/pg1‚4.030620452363502e-05 & lr-AdamW/pg2‚4.030620452363502e-05 trainer/global_step‚10449 $ _timestamp‚1.788537176108394e+09 _runtime‚12559.643949537 _step‚856Ø+¤l4Š+/ _wandbruntime‚12559  _runtime‚12559«JMù(‹+P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264658093 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚892259708486656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130905548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231750656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚971238449030656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚12559.64481178 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11925082325920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11969835665920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264658093 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264631749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚892260636461056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚892259708486656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264631749 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924174350916 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863094628964276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863095553978640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880290800795136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12498715236864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚857 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880290782792192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218482696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 483140791 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207810560 % _timestamp‚1.7885371761091878e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218482698 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚892260636461056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218482696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63339235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚851170454613360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚851170471652720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880290800795136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63338596 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚983737188207616½}÷ (Œ+†P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 483140791 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207810560 % _timestamp‚1.7885371761091878e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218482698 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚892260636461056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218482696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63339235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚851170454613360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚851170471652720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880290800795136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63338596 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚983737188207616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264658093 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚892259708486656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130905548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231750656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚971238449030656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚12559.64481178 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11925082325920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11969835665920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264658093 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264631749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚892260636461056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚892259708486656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264631749 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924174350916 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863094628964276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863095553978640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880290800795136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12498715236864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚857 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880290782792192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218482696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944ÙÞ.¦4+/ _wandbruntime‚12560  _runtime‚12560à¼oì Ž+ŒN K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264656580 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926199599406 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880372419880960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚892343377020416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194261969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863175688846670 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E&mall_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11970047154688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63344358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925291624194 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880372401878016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11970957139456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚851249472207904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863174763832098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218502856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194262612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚983828594871808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚971328674735104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 264682950 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11970957139456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12499896196608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 483185808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 483159436 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851249489247264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264656580 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚10449 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218502858 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12499667427840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚983828342162944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚971328698675200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130917613 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218502858WãO'+‘N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚880372401878016 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130917611 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 483185808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _runtime‚12560.742727038 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚858 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 483159436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63344999 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚892342449032704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11970047154688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63344358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925291624194 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880372401878016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11970957139456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚851249472207904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863174763832098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218502856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194262612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚983828594871808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚971328674735104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 264682950 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11970957139456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12499896196608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 483185808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 483159436 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851249489247264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264656580 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚10449 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218502858 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12499667427840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚983828342162944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚971328698675200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130917613 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218502858 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264656580 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926199599406 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880372419880960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚892343377020416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194261969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863175688846670 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896396750202880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚10499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚884371737083392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12025923090944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11980959581028 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11980051606024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚861 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265876651 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚855121351379920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12025923090944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚884371737083392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.oÞqÂã n_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131508796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988306965059584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896397678177280 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12025013119488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131508798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63631444 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚855121334340560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884371755086336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485393693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12557457491456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867101385946584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Xx…y (ž+†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 485367347 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896396750202880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚10499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚884371737083392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12025923090944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11980959581028 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11980051606024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚861 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265876651 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚855121351379920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12025923090944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚884371737083392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131508796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988306965059584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896397678177280 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12025013119488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131508798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63631444 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚855121334340560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884371755086336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485393693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12557457491456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867101385946584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219490696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚884371755086336 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195139598 _runtime‚12614.387839034 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚896397678177280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12025013119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265902995 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265876651 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚896396750202880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63630802 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788537230851894e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219490698 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219490698 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 219490696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265902995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚975749758193152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚975749734253056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ÝfJ¦”Ÿ+‚(:‰ ÓëÔ°âá gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¾ +‚(:ö ÓëÔ¡ëµã disk.sdb2.in‚ 75.59765625 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375 network.recv‚ 4.198948e+06) disk./.usagePercent‚76.06202339807827 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.md0p1.out‚ 1812.5390625 disk.md0.in‚ 208.765625" disk.sdb2.out‚1862.99853515625! proc.memory.rssMB‚ 2902.359375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7674.3046875" disk.sda2.out‚1862.99853515625! disk.sda.out‚1862.99853515625# disk.nvme1n1.out‚62654.33203125 disk.rbd0.in‚0$ memory_percent‚91.55759811401367* proc.memory.percent‚17.714595794677734 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625! disk.nvme0n1.out‚ 144529.46875 disk.md0.out‚ 1812.5390625 cpu‚25.21992142000221 proc.cpu.threads‚17 network.sent‚ 2.8064775e+07 disk.nvme1n1.in‚ 850.26171875 disk.dm-0.in‚ 8524.56640625! disk.dm-0.out‚207183.80078125 disk.sdb.in‚ 75.59765625' proc.memory.availableMB‚ 1383.203125% disk./.usageGB‚2720.3463821411133 disk.sda1.in‚0! disk.sdb.out‚1862.99853515625rïW4¡+/ _wandbruntime‚12615  _runtime‚12615N¹›4'¢+ŒN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485438710 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚862 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485412338 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚884453374172160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12026134579712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485438710 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11982076854514 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884453356169216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12027044564480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12558414631424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚867182445828978 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12026134579712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚12615.712346758 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195157396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63637165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988398374588928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885372321743429e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219510858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚975839959957504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265901482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚896480418736640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896480418736640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚975839983897600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12027044564480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚855200368974464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265927852 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195158028 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265901482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚10499 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784®/(â…£+‘N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485438710 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚862 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485412338 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚884453374172160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12026134579712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485438710 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11982076854514 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884453356169216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12027044564480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12558414631424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚867182445828978 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12026134579712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚12615.712346758 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195157396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63637165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988398374588928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885372321743429e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219510858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚975839959957504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265901482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚896480418736640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896480418736640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚975839983897600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12027044564480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚855200368974464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265927852 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195158028 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265901482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚10499 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784Þ/0È4¤+/ _wandbruntime‚12615  _runtime‚12615D<‹T¥+Î /loss/FRt1/assignment_loss‚2.332199763684046 /loss/FRt2/assignment_loss‚3.127519399889054 /loss/FRt1/detection_loss‚3.2545327714511325 0loss/SRqqt1/detection_loss‚2.168919281235763 _step‚863 .loss/FRt4/assignment_loss‚2.87269731760025 2loss/SRqqt4/assignment_loss‚1.4543216607787393 1loss/FBt2/assignment_loss‚0.17296752320768205 1loss/SRqqt3/detection_loss‚0.7320770982652902 .loss/FRt3/assignment_loss‚0.97256367901961 1loss/FBt3/assignment_loss‚0.04569047308461454 .loss/FRt3/detection_loss‚1.364126096231242 /loss/FBt2/detection_loss‚2.4130294162493486 % _timestamp‚1.7885372321794071e+09 1loss/FBt4/assignment_loss‚0.27348323729320967 0loss/SRqqt2/detection_loss‚2.589414162480313 .loss/FBt3/detection_loss‚0.438779279339786 trainer/global_step‚10499 1loss/SRqqt1/assignment_loss‚1.470494972275836 1loss/FBt1/assignment_loss‚0.17512209278604018 .loss/FRt2/detection_loss‚2.339189373085373 $loss/total_loss‚39.5178572940019 .loss/FRt4/detection_loss‚2.294996439712122 .loss/FBt1/detection_loss‚2.409321568906307 _runtime‚ 12615.7149664 2loss/SRqqt2/assignment_loss‚1.5059359308494173 2loss/SRqqt3/assignment_loss‚0.8826991251460277 1loss/SRqqt4/detection_loss‚2.0894129068562477 /loss/FBt4/detection_loss‚2.1383637245744467 epoch‚8YgR)Y¦+Ó /loss/FRt1/assignment_loss‚2.332199763684046 /loss/FRt2/assignment_loss‚3.127519399889054 /loss/FRt1/detection_loss‚3.2545327714511325 0loss/SRqqt1/detection_loss‚2.168919281235763 _step‚863 .loss/FRt4/assignment_loss‚2.87269731760025 2loss/SRqqt4/assignment_loss‚1.4543216607787393 1loss/FBt2/assignment_loss‚0.17296752320768205 1loss/SRqqt3/detection_loss‚0.7320770982652902 .loss/FRt3/assignment_loss‚0.97256367901961 1loss/FBt3/assignment_loss‚0.04569047308461454 .loss/FRt3/detection_loss‚1.364126096231242 /loss/FBt2/detection_loss‚2.4130294162493486 % _timestamp‚1.7885372321794071e+09 1loss/FBt4/assignment_loss‚0.27348323729320967 0loss/SRqqt2/detection_loss‚2.589414162480313 .loss/FBt3/detection_loss‚0.438779279339786 trainer/global_step‚10499 1loss/SRqqt1/assignment_loss‚1.470494972275836 1loss/FBt1/assignment_loss‚0.17512209278604018 .loss/FRt2/detection_loss‚2.339189373085373 $loss/total_loss‚39.5178572940019 .loss/FRt4/detection_loss‚2.294996439712122 .loss/FBt1/detection_loss‚2.409321568906307 _runtime‚ 12615.7149664 2loss/SRqqt2/assignment_loss‚1.5059359308494173 2loss/SRqqt3/assignment_loss‚0.8826991251460277 1loss/SRqqt4/detection_loss‚2.0894129068562477 /loss/FBt4/detection_loss‚2.1383637245744467 epoch‚8߈ ”§+‚(:‰ žÓëÔ˜ ±ß gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ë…¨+‚(:ú žÓëԥɲá network.sent‚ 2.8161161e+07$ disk.nvme0n1.out‚144577.77734375" disk.sda2.out‚1864.48095703125" disk.sdb2.out‚1864.48095703125# memory_percent‚91.5592908859253) disk./.usagePercent‚76.05949842330135 disk.sdb2.in‚ 75.59765625 proc.cpu.threads‚17$ disk./.usageGB‚2720.256076812744! disk.sdb.out‚1864.48095703125 cpu‚25.21966906673675 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7674.3046875 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8524.56640625 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1382.92578125! proc.memory.rssMB‚ 2902.359375 network.recv‚ 4.205217e+06 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.md0p1.out‚ 1813.96484375 disk.rbd0.out‚ 3.87109375 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.md0.out‚ 1813.96484375! disk.sda.out‚1864.48095703125 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0 disk.rbd2.out‚ 3.87109375 disk.nvme1n1.in‚ 850.26171875" disk.nvme1n1.out‚ 62655.5546875 disk.sda1.out‚0* proc.memory.percent‚17.714595794677734! disk.dm-0.out‚207233.33203125«—?C”©+‚(:‰ ­ÓëÔ«³ÿÞ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü'T€ª+‚(:õ ­ÓëÔæýÔá network.sent‚ 2.8245871e+07 disk.md0.in‚ 208.765625$ memory_percent‚91.55886173248291$ disk./.usageGB‚2720.392219543457! disk.sda.out‚1866.42822265625 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375" disk.sdb2.out‚1866.42822265625 disk.sdb.in‚ 75.59765625) disk./.usagePercent‚76.06330503103224 disk.md0p1.in‚ 208.765625 disk.nvme0n1.in‚ 7674.3046875 cpu‚25.219407885088113! disk.nvme1n1.out‚ 62655.734375 disk.sdb1.out‚0 disk.sdb1.in‚0! proc.memory.rssMB‚ 2902.359375! disk.sdb.out‚1866.42822265625 network.recv‚ 4.212231e+06 disk.md0p1.out‚ 1815.85546875 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.out‚ 144688.0625 disk.md0.out‚ 1815.85546875* proc.memory.percent‚17.714595794677734 proc.cpu.threads‚17 disk.dm-0.out‚ 207343.796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0" disk.sda2.out‚1866.42822265625 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8524.56640625) proc.memory.availableMB‚ 1382.99609375êÎô ”«+‚(:‰ ¼ÓëÔù”Çà gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¶¢²ƒ¬+‚(:ø ¼ÓëÔàåˆâ# disk.nvme1n1.out‚62656.41015625 disk.md0.out‚ 1817.33984375) proc.memory.availableMB‚ 1383.25390625! disk.dm-0.out‚207618.21484375 disk.sdb.in‚ 75.59765625! disk.sdb2.out‚1867.9697265625 disk.md0p1.in‚ 208.765625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7674.3046875# disk.nvme0n1.out‚144961.8046875 disk.sdb.out‚1867.9697265625 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8524.56640625 disk.sdb1.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.714595794677734 cpu‚25.219146762735384 disk.md0p1.out‚ 1817.33984375 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875! disk.sda2.out‚1867.9697265625$ memory_percent‚91.55728816986084 proc.cpu.threads‚17 network.sent‚ 2.8254414e+07) disk./.usagePercent‚76.05580295584492 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚ 3.87109375! proc.memory.rssMB‚ 2902.359375 network.recv‚ 4.216903e+06 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2720.123908996582 disk.sda.in‚ 133.16796875 disk.sda.out‚1867.9697265625Å}‹4­+/ _wandbruntime‚12671  _runtime‚12671ºÀÜôÇ®+Á % lr-AdamW/pg1‚3.56224186366212e-05 % lr-AdamW/pg2‚3.56224186366212e-05 trainer/global_step‚10549 % _timestamp‚1.7885372875839713e+09 _runtime‚12671.119663421 _step‚8643CL˜Ì¯+Æ % lr-AdamW/pg2‚3.56224186366212e-05 trainer/global_step‚10549 % _timestamp‚1.7885372875839713e+09 _runtime‚12671.119663421 _step‚864 % lr-AdamW/pg1‚3.56224186366212e-05à¾Î¹4°+/ _wandbruntime‚12671  _runtime‚12671s¸+Ö(±+‚P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888452691374592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132112048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859072214067760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚10549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220498698 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 267147897 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267121553 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12081100544512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888452709377536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63923514 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888452709377536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12035928861132 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12082010515968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885372875851917e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196034921 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888452691374592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12082010515968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487646595 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220498698 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚992877187321856 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267121553 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚900534719893504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 267147897 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12616143906304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12615915124224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859072231107120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132112046 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900534719893504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487620249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220498696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚992876934599680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900533791919104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220498696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚980261043415552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196035562 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63922875 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚980261019475456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚865 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871108142928892~ ¤- (²+‡P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12615915124224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859072231107120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132112046 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900534719893504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487620249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220498696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚992876934599680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900533791919104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220498696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚980261043415552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196035562 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63922875 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚980261019475456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚865 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871108142928892 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888452691374592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132112048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859072214067760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚10549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220498698 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 267147897 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267121553 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12081100544512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888452709377536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63923514 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888452709377536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12035928861132 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12082010515968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885372875851917e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196034921 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888452691374592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12082010515968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487646595 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220498698 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚992877187321856 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267121553 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚900534719893504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 267147897 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12616143906304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0á?Þ4³+/ _wandbruntime‚12672  _runtime‚12672,”·´+ŒN MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12083131989504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12082222004736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888534328463360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267146384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900617460452864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220518858 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888534310460416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871189202811286 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196052762 H@DeviceStatsMol«Ö‹nitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487665240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487691612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267146384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888534328463360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚866 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487691612 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196053400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚992968325373440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63928651 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132124111 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12617080193536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚900616532465152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚992968578082304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚859151248701664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220518856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888534310460416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871188277796714 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚900617460452864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132124113 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570OëÜT'µ+‘N trainer/global_step‚10549 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220518858 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132124111 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12617080193536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚900616532465152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚992968578082304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚859151248701664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220518856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888534310460416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871188277796714 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚900617460452864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132124113 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12083131989504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12082222004736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888534328463360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267146384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900617460452864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220518858 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888534310460416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871189202811286 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196052762 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487665240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487691612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267146384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888534328463360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚866 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487691612 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196053400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚992968325373440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63928651 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FŠ4¶+/ _wandbruntime‚12672  _runtime‚12672æÒµº[·+Õ %loss/total_loss‚39.53605259619862 .loss/FRt2/detection_loss‚2.689169300059651 /loss/FBt1/detection_loss‚2.4948754335443177 _runtime‚12672.337704079 /loss/FRt1/detection_loss‚2.9097527583440144 2loss/SRqqt4/assignment_loss‚1.4279859193214557 0loss/FBt2/assignment_loss‚0.2681444130546879 0loss/SRqqt4/detection_loss‚2.155843917189575 1loss/FBt4/assignment_loss‚0.07664353107098615 .loss/FRt3/detection_loss‚1.462831653418049 1loss/SRqqt3/detection_loss‚0.6880946759626551 % _timestamp‚1.7885372888021576e+09 0loss/FRt4/assignment_loss‚3.2135035190088996 0loss/FBt1/assignment_loss‚0.1258320705153627 /loss/FBt2/detection_loss‚2.2765924096107484 trainer/global_step‚10549 2loss/SRqqt3/assignment_loss‚0.9293560654118236 .loss/FRt2/assignment_loss‚2.67210919658343 1loss/SRqqt1/assignment_loss‚1.385234058129065 2loss/SRqqt2/assignment_loss‚1.4593307583739883 0loss/SRqqt2/detection_loss‚2.252614162470165 /loss/FBt3/detection_loss‚0.5953061400850614 _step‚867 /loss/FRt1/assignment_loss‚3.025146027406057 0loss/FRt3/assignment_loss‚0.8117583346745324 /loss/FRt4/detection_loss‚2.2353529030906745 epoch‚8 2loss/FBt3/assignment_loss‚0.054688094360752094 1loss/SRqqt1/detection_loss‚2.2468970652782554 .loss/FBt4/detection_loss‚2.078990189234416Rm‹`¸+Ú 2loss/SRqqt4/assignment_loss‚1.4279859193214557 0loss/FBt2/assignment_loss‚0.2681444130546879 0loss/SRqqt4/detection_loss‚2.155843917189575 1loss/FBt4/assignment_loss‚0.07664353107098615 .loss/FRt3/detection_loss‚1.462831653418049 1loss/SRqqt3/detection_loss‚0.6880946759626551 % _timestamp‚1.7885372888021576e+09 0loss/FRt4/assignment_loss‚3.2135035190088996 0loss/FBt1/assignment_loss‚0.1258320705153627 /loss/FBt2/detection_loss‚2.2765924096107484 trainer/global_step‚10549 2loss/SRqqt3/assignment_loss‚0.9293560654118236 .loss/FRt2/assignment_loss‚2.67210919658343 1loss/SRqqt1/assignment_loss‚1.385234058129065 2loss/SRqqt2/assignment_loss‚1.4593307583739883 0loss/SRqqt2/detection_loss‚2.252614162470165 /loss/FBt3/detection_loss‚0.5953061400850614 _step‚867 /loss/FRt1/assignment_loss‚3.025146027406057 0loss/FRt3/assignment_loss‚0.8117583346745324 /loss/FRt4/detection_loss‚2.2353529030906745 epoch‚8 2loss/FBt3/assignment_loss‚0.054688094360752094 1loss/SRqqt1/detection_loss‚2.2468970652782554 .loss/FBt4/detection_loss‚2.078990189234416 %loss/total_loss‚39.53605259619862 .loss/FRt2/detection_loss‚2.689169300059651 /loss/FBt1/detection_loss‚2.4948754335443177 _runtime‚12672.337704079 /loss/FRt1/detection_loss‚2.9097527583440144ãwÖò“¹+‚(:ˆ ËÓëÔÀÚà gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éß9†º+‚(:û ËÓëÔþ Ãá disk.sda1.in‚0! disk.sdb.out‚1869.43798828125 cpu‚25.218909785402357 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 8524.56640625! disk.dm-0.out‚207670.70703125 proc.cpu.threads‚17 network.recv‚ 4.220513e+06 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0" disk.sda2.out‚1869.43798828125 disk.md0.in‚ 208.765625* proc.memory.percent‚17.714595794677734 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1869.43798828125# disk.nvme1n1.out‚62656.62890625 disk.nvme0n1.in‚ 7674.3046875 disk.md0.out‚ 1818.73828125 disk.sdb.in‚ 75.59765625! disk.sda.out‚1869.43798828125! proc.memory.rssMB‚ 2902.359375 disk.rbd2.in‚0" disk.nvme0n1.out‚ 145014.078125 disk.md0p1.out‚ 1818.73828125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda.in‚ 133.16796875$ memory_percent‚91.55793190002441 network.sent‚ 2.8270439e+07) disk./.usagePercent‚76.05824782856222 disk.nvme1n1.in‚ 850.26171875( proc.memory.availableMB‚ 1383.1484375% disk./.usageGB‚2720.2113494873047×.;˜”»+‚(:‰ ÚÓëÔžÚ˜à gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;º¼+‚(:ö ÚÓëÔ­—•á disk.sdb1.in‚0 disk.sdb.in‚ 75.59765625 disk.rbd2.out‚ 3.87109375 disk.md0p1.out‚ 1820.22265625 network.recv‚ 4.227332e+06 disk.sdb1.out‚0 disk.nvme0n1.in‚ 7674.3046875 disk.sdb2.in‚ 75.59765625" disk.nvme0n1.out‚ 145066.296875 disk.md0.out‚ 1820.22265625 disk.dm-0.in‚ 8524.56640625 disk.md0p1.in‚ 208.765625* proc.memory.percent‚17.714595794677734! disk.sda2.out‚1870.9794921875 disk.sda1.in‚0 disk.sda.in‚ 133.16796875 disk.sda.out‚1870.9794921875 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0! proc.memory.rssMB‚ 2902.359375 cpu‚25.218672688075465 disk.sda1.out‚0 disk.md0.in‚ 208.765625! disk.dm-0.out‚207783.67578125$ memory_percent‚91.55640602111816) disk./.usagePercent‚76.06344550295304$ disk./.usageGB‚2720.397243499756 disk.rbd2.in‚0 disk.sdb.out‚1870.9794921875! disk.sdb2.out‚1870.9794921875# disk.nvme1n1.out‚62717.37890625 disk.rbd0.out‚ 3.87109375( proc.memory.availableMB‚ 1383.3984375 network.sent‚ 2.8365223e+07 disk.sda2.in‚ 133.16796875 proc.cpu.threads‚17Ý8(¸”½+‚(:‰ éÓëÔúË·à gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d£C€¾+‚(:õ éÓëÔõãÑà network.sent‚ 2.8371492e+07 disk.nvme0n1.in‚ 7674.3046875 disk.sda.out‚1872.8759765625 disk.md0p1.in‚ 208.765625! disk.sda2.out‚1872.8759765625$ memory_percent‚91.55757427215576! proc.memory.rssMB‚ 2902.359375 disk.dm-0.in‚ 8524.56640625 disk.sdb1.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚145273.60546875 disk.sda2.in‚ 133.16796875 proc.cpu.threads‚17 network.recv‚ 4.232011e+06! disk.sdb2.out‚1872.8759765625" disk.nvme1n1.out‚ 62781.6171875 disk.rbd2.out‚ 3.87109375* proc.memory.percent‚17.714595794677734! disk.dm-0.out‚208055.22265625 disk.sdb.out‚1872.8759765625 disk.sdb2.in‚ 75.59765625 disk.sdb1.in‚0 disk.rbd2.in‚0% disk./.usageGB‚2720.7715911865234 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1822.046875 disk.rbd0.out‚ 3.87109375 disk.md0.in‚ 208.765625 cpu‚25.218456486998935) disk./.usagePercent‚76.07391242095176 disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.md0.out‚ 1822.046875) proc.memory.availableMB‚ 1383.20703125`ø”¿+‚(:‰ øÓëÔôŸ”ß gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.213' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.05325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùj=uÀ+‚(:ê øÓëÔ†»…á! disk.dm-0.out‚208209.00390625 disk.sda1.in‚0) disk./.usagePercent‚76.07742955199845 disk.sdb1.in‚0 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8524.56640625 disk.sda.out‚ 1874.3203125! proc.memory.rssMB‚ 2902.359375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375 disk.sda2.out‚ 1874.3203125* proc.memory.percent‚17.714595794677734 disk.sda.in‚ 133.16796875 disk.rbd2.in‚0 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 7674.3046875$ disk.nvme0n1.out‚145377.99609375" disk.nvme1n1.out‚ 62831.0078125% disk./.usageGB‚2720.8973808288574$ memory_percent‚91.56057834625244 disk.rbd0.in‚0 disk.md0.out‚ 1823.4296875 network.sent‚ 2.8376744e+07 disk.sdb.in‚ 75.59765625 disk.sdb.out‚ 1874.3203125 disk.sdb2.in‚ 75.59765625 disk.sdb2.out‚ 1874.3203125 disk.nvme1n1.in‚ 850.26171875 network.recv‚ 4.235385e+06 disk.md0p1.out‚ 1823.4296875 disk.sdb1.out‚0) proc.memory.availableMB‚ 1382.71484375 cpu‚25.21823696052017 disk.md0.in‚ 208.765625î:Ž4Á+/ _wandbruntime‚12729  _runtime‚12729£Ž!ÛÈÂ+ & lr-AdamW/pg2‚3.338352917786811e-05 trainer/global_step‚10599 $ _timestamp‚1.788537346045612e+09 _runtime‚12729.581162552 _step‚868 & lr-AdamW/pg1‚3.338352917786811e-05 í#LÍÃ+Ç & lr-AdamW/pg2‚3.338352917786811e-05 trainer/global_step‚10599 $ _timestamp‚1.788537346045612e+09 _runtime‚12729.581162552 _step‚868 & lr-AdamW/pg1‚3.338352917786811e-05ä³ ©É4Ä+/ _wandbruntime‚12729  _runtime‚12729@ù\›©Å+‚P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221506696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12092714091244 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚10599 _runtime‚12729.582044956 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221506698 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268392799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚892533663668736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892533645665792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196931118 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885373460463948e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚904670833635328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997447047248384 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12137187969536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489899497 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132715298 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12138097940992 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132715296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221506698 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚984772328637952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚863023110834320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221506696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12138097940992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚869 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚904671761609728 LBDeviceStatsMonitor.on_train_batch_start/alloca¬6cy_tion.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196930495 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚904670833635328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268392799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875114899911200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12137187969536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 268366455 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚904671761609728 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875115824925564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64215199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12674742550528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489873151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64215820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863023093794960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12091806116240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997447293679104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚984772304697856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12674965041152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004Úv¯ (Æ+‡P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892533645665792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196931118 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885373460463948e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚904670833635328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997447047248384 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12137187969536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489899497 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132715298 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12138097940992 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132715296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221506698 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚984772328637952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚863023110834320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221506696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12138097940992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚869 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚904671761609728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196930495 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚904670833635328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268392799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875114899911200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12137187969536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 268366455 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚904671761609728 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875115824925564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64215199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12674742550528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489873151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64215820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863023093794960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12091806116240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997447293679104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚984772304697856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12674965041152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221506696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12092714091244 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚10599 _runtime‚12729.582044956 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221506698 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268392799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚892533663668736åŠ~\4Ç+/ _wandbruntime‚12730  _runtime‚12730¦_''È+‹N PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221526858 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138309429760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221526856 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132727361 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875195959793594 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚12730.760939701 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 221526858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚892615264751616 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196948257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997538696952320 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489944514 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196948883 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚984862554342400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12676142609920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863102111389504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚904753574181376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984862530402304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 trainer/global_step‚10599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268417656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 $ _timestamp‚1.788537347224642e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚904754502169088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489944514 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268391286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚904754502169088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268417656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64220896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132727363 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12092923389518®HøÐ,É+N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875195034779022 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 268391286 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚870 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12139219414528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚892615264751616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚892615282754560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489918142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 489918142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚997538452632064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚12730.760939701 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 221526858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚892615264751616 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196948257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997538696952320 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489944514 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196948883 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚984862554342400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12676142609920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863102111389504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚904753574181376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984862530402304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 trainer/global_step‚10599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268417656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 $ _timestamp‚1.788537347224642e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚904754502169088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489944514 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268391286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚904754502169088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268417656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64220896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132727363 UEDeviceStatsG9~vê Monitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12092923389518 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221526858 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138309429760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221526856 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132727361 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875195959793594 Eh”Ó+‚(:‰ ´ÔëÔÊÿþÞ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò8Ók‚Ô+‚(:÷ ´ÔëÔ÷‹ùß disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7674.3046875) proc.memory.availableMB‚ 1382.90234375! proc.memory.rssMB‚ 2902.359375 disk.md0p1.in‚ 208.765625# disk.nvme0n1.out‚145896.6328125! disk.nvme1n1.out‚ 63158.171875 disk.sdb1.out‚0 disk.sda1.in‚0 network.sent‚ 2.8502322e+07 network.recv‚ 4.254733e+06( disk./.usagePercent‚76.0827073284489$ disk./.usageGB‚2721.086139678955 disk.md0.in‚ 208.765625 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1829.58203125! disk.sdb.out‚1880.73486328125 disk.nvme1n1.in‚ 850.26171875* proc.memory.percent‚17.714595794677734 cpu‚25.21739447082937 proc.cpu.threads‚17 disk.sdb1.in‚0$ memory_percent‚91.55943393707275 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚1880.73486328125! disk.sda.out‚1880.73486328125 disk.md0p1.out‚ 1829.58203125 disk.sda1.out‚0" disk.sda2.out‚1880.73486328125 disk.dm-0.in‚ 8524.56640625 disk.sdb.in‚ 75.59765625 disk.dm-0.out‚209054.8046875ÙMB“4Õ+/ _wandbruntime‚12788  _runtime‚12788F©ÉÖ+à & lr-AdamW/pg1‚3.121393559853508e-05 & lr-AdamW/pg2‚3.121393559853508e-05 trainer/global_step‚10649 % _timestamp‚1.7885374048643277e+09 _runtime‚12788.400096984 _step‚872ÔžA¿Î×+È _step‚872 & lr-AdamW/pg1‚3.121393559853508e-05 & lr-AdamW/pg2‚3.121393559853508e-05 trainer/global_step‚10649 % _timestamp‚1.7885374048643277e+09 _runtime‚12788.400096984èYÿ' 4Ø+/ _wandbruntime‚12788  _runtime‚12788û"Óñ (Ù+ƒP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269637701 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269611357 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 222514698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚896614599956992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989283613860352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885374048655043e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚873 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908808803325952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 269611357 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222514696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908807875351552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194185366016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚896614599956992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201519104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879121656893508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133318546 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492152399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908807875351552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12733482583040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866973990561520 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1002017297962496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989283589920256 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 492126053 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896614617959936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194185366016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64507343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879122581907872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12193275394560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225459200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222514696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908808803325952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚896614617959936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12193275394560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269637701 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ú6ëP(Ú+ˆP VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908808803325952 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 269611357 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222514696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908807875351552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194185366016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚896614599956992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201519104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879121656893508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133318546 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492152399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908807875351552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12733482583040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866973990561520 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1002017297962496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989283589920256 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 492126053 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896614617959936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194185366016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64507343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879122581907872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12193275394560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225459200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222514696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908808803325952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚896614617959936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12193275394560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269637701 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269637701 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269611357 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 222514698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚896614599956992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989283613860352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885374048655043e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚873éríýž(Û+/ _wandbruntime‚12789  _runS Œ¹ time‚12789`l¦'Ü+N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12195306839552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222534856 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1002108707672064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197843687 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269636188 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896696219042816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12194396854784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 492171044 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492197416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚908890615897600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12195306839552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867053008156064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12734868107264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222534858 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚896696237045760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚989373839564800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269636188 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚989373815624704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241351680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879202716775902 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚12789.556785319 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64512419 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867052991116704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12148800644626 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 492171044 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12149708619838 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚908891543885312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1002108442380288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879201791761330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 492197416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12734626755584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269662558 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197843030 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269662558 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12194396854784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64512419 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867052991116704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12148800644626 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 492171044 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12149708619838 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚908891543885312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1002108442380288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879201791761330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 492197416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12734626755584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269662558 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197843030 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269662558 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12194396854784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197843687 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269636188 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896696219042816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12194396854784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 492171044 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492197416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760êøoï4Þ+/ _wandbruntime‚12789  _runtime‚12789ùr³Qbß+Ü 0loss/FRt2/assignment_loss‚2.8289782921473186 1loss/FBt4/assignment_loss‚0.00439650192856315 0loss/FRt3/assignment_loss‚0.8835794412172757 2loss/SRqqt4/assignment_loss‚1.5311609606861192 .loss/FBt3/detection_loss‚0.542077931411126 0loss/FRt1/assignment_loss‚3.3409744912180406 2loss/SRqqt1/assignment_loss‚1.5213748397249165 .loss/FRt2/detection_loss‚2.548525001639261 1loss/SRqqt3/detection_loss‚0.7967830113887501 .loss/FBt2/detection_loss‚2.329384066841819 _step‚875 1loss/SRqqt2/assignment_loss‚1.458457063523031 2loss/FBt3/assignment_loss‚0.052860828205559415 /loss/FRt3/detection_loss‚1.2938474937700308 /loss/FRt4/detection_loss‚1.9890023500863778 0loss/FRt4/assignment_loss‚3.1003002716323076 3loss/FBt2/assignment_loss‚0.0009315236873748942 /loss/FRt1/detection_loss‚2.9527618679507026 1loss/SRqqt1/detection_loss‚1.9797342627337484 /loss/FBt4/detection_loss‚2.8402867913246155 %loss/total_loss‚40.89770270545107 2loss/SRqqt3/assignment_loss‚0.9353348425885577 epoch‚8 0loss/SRqqt2/detection_loss‚2.505330640462137 /loss/FBt1/detection_loss‚3.0412336587905884 1loss/FBt1/assignment_loss‚0.20349867418796444 0loss/SRqqt4/detection_loss‚2.216887898304883 trainer/global_step‚10649 % _timestamp‚1.7885374060234323e+09 _runtime‚12789.559405422‚‡ÈÆgà+á _step‚875 1loss/SRqqt2/assignment_loss‚1.458457063523031 2loss/FBt3/assignment_loss‚0.052860828205559415 /loss/FRt3/detection_loss‚1.2938474937700308 /loss/FRt4/detection_loss‚1.9890023500863778 0loss/FRt4/assignment_loss‚3.1003002716323076 3loss/FBt2/assignment_loss‚0.0009315236873748942 /loss/FRt1/detection_loss‚2.9527618679507026 1loss/SRqqt1/detection_loss‚1.9797342627337484 /loss/FBt4/detection_loss‚2.8402867913246155 %loss/total_loss‚40.89770270545107 2loss/SRqqt3/assignment_loss‚0.9353348425885577 epoch‚8 0loss/SRqqt2/detection_loss‚2.505330640462137 /loss/FBt1/detection_loss‚3.0412336587905884 1loss/FBt1/assignment_loss‚0.20349867418796444 0loss/SRqqt4/detection_loss‚2.216887898304883 trainer/global_step‚10649 % _timestamp‚1.7885374060234323e+09 _runtime‚12789.559405422 0loss/FRt2/assignment_loss‚2.8289782921473186 1loss/FBt4/assignment_loss‚0.00439650192856315 0loss/FRt3/assignment_loss‚0.8835794412172757 2loss/SRqqt4/assignment_loss‚1.5311609606861192 .loss/FBt3/detection_loss‚0.542077931411126 0loss/FRt1/assignment_loss‚3.3409744912180406 2loss/SRqqt1/assignment_loss‚1.5213748397249165 .loss/FRt2/detection_loss‚2.548525001639261 1loss/SRqqt3/detection_loss‚0.7967830113887501 .loss/FBt2/detection_loss‚2.329384066841819ë å}’á+‚(:‡ ÃÔëÔæêšà gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç$B„â+‚(:ù ÃÔëÔчçà disk.md0.in‚ 208.765625! proc.memory.rssMB‚ 2902.359375 disk.rbd2.in‚0! disk.sda.out‚1882.25634765625 disk.md0p1.in‚ 208.765625$ memory_percent‚91.55819416046143! disk.nvme0n1.in‚ 7676.55078125 cpu‚25.21717561674921# disk.nvme0n1.out‚146000.2890625 disk.sda.in‚ 133.16796875 disk.sda1.out‚0* proc.memory.percent‚17.714595794677734% disk./.usageGB‚2721.1469116210938 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚63220.69140625 disk.md0p1.out‚ 1831.04296875 disk.rbd0.out‚ 3.87109375 network.sent‚ 2.8507709e+07 disk.md0.out‚ 1831.04296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 4.258886e+06 disk.rbd2.out‚ 3.87109375! disk.dm-0.out‚209220.98046875! disk.sdb.out‚1882.25634765625 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0) disk./.usagePercent‚76.08440653738614 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8526.8125" disk.sdb2.out‚1882.25634765625) proc.memory.availableMB‚ 1383.10546875" disk.sda2.out‚1882.25634765625wG¼“ã+‚(:ˆ ÒÔëÔæžÏÞ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ?§„ä+‚(:ö ÒÔëÔä·ðß! disk.nvme0n1.in‚ 7676.55078125 disk.sdb.in‚ 75.59765625! disk.sdb2.out‚1884.3505859375 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1382.36328125! proc.memory.rssMB‚ 2902.359375 cpu‚25.216983977071962 network.sent‚ 2.8617482e+07 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375! disk.sda2.out‚1884.3505859375 proc.cpu.threads‚17! disk.dm-0.out‚209369.48828125 disk.sdb1.in‚0 disk.sda1.out‚0% disk./.usageGB‚2721.2721977233887 disk.sdb.out‚1884.3505859375 disk.sda.out‚1884.3505859375 disk.md0p1.out‚ 1833.07421875 disk.sdb1.out‚0$ disk.nvme0n1.out‚146104.07421875 disk.md0.out‚ 1833.07421875$ memory_percent‚91.56272411346436 network.recv‚ 4.265756e+06" disk.nvme1n1.out‚ 63265.4140625 disk.md0.in‚ 208.765625) disk./.usagePercent‚76.08790958924261 disk.sdb2.in‚ 75.59765625 disk.sda2.in‚ 133.16796875* proc.memory.percent‚17.714595794677734 disk.dm-0.in‚ 8526.8125 disk.sda1.in‚06Ï°á”å+‚(:‰ áÔëÔ¢çñÞ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë(Ê}æ+‚(:ò áÔëÔîäƒâ# disk.nvme1n1.out‚63398.66796875 disk.sda1.in‚0! disk.sdb2.out‚1885.6259765625 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sda2.out‚1885.6259765625 disk.sdb.out‚1885.6259765625 disk.sdb2.in‚ 75.59765625 disk.sda1.out‚0 disk.rbd2.out‚ 3.87109375 disk.sda.out‚1885.6259765625 cpu‚25.216760813354202 network.recv‚ 4.270352e+06 disk.rbd0.in‚0 disk.sda2.in‚ 133.16796875% disk./.usageGB‚2721.3968544006348 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7676.55078125! disk.dm-0.out‚209658.22265625 disk.sdb.in‚ 75.59765625 network.sent‚ 2.8624247e+07 disk.md0p1.in‚ 208.765625 disk.sda.in‚ 133.16796875) proc.memory.availableMB‚ 1382.87890625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1834.296875# disk.nvme0n1.out‚146259.5546875 disk.md0.in‚ 208.765625 disk.md0.out‚ 1834.296875 disk.dm-0.in‚ 8526.8125! proc.memory.rssMB‚ 2902.359375) disk./.usagePercent‚76.09139504211129$ memory_percent‚91.55957698822021* proc.memory.percent‚17.714595794677734ï}.‘ç+‚(:† ðÔëÔâü‹á gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Ö=Çxè+‚(:í ðÔëÔ…Ÿ¦á disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 proc.cpu.threads‚17! disk.sdb.out‚1887.63916015625 disk.sdb2.in‚ 75.59765625 disk.md0p1.out‚1836.25 disk.rbd0.out‚ 3.87109375 disk.md0.out‚1836.25 cpu‚25.216530457657036 disk.md0.in‚ 208.765625! disk.dm-0.out‚209829.14453125 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875! proc.memory.rssMB‚ 2902.359375 disk.dm-0.in‚ 8526.8125! disk.nvme1n1.out‚ 63465.609375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875" disk.sdb2.out‚1887.63916015625! disk.sda.out‚1887.63916015625 network.sent‚ 2.8629929e+07) disk./.usagePercent‚76.09139664201928 disk.sda1.out‚0" disk.sda2.out‚1887.63916015625$ memory_percent‚91.56177043914795 network.recv‚ 4.274762e+06 disk.rbd2.out‚ 3.87109375! disk.nvme0n1.in‚ 7676.55078125$ disk.nvme0n1.out‚146363.53515625) proc.memory.availableMB‚ 1382.51953125* proc.memory.percent‚17.714595794677734% disk./.usageGB‚2721.3969116210938 disk.sdb1.out‚0ÉIyÔ4é+/ _wandbruntime‚12846  _runtime‚12846ed¤Êê+Ä ' lr-AdamW/pg1‚2.9114090038672216e-05 ' lr-AdamW/pg2‚2.9114090038672216e-05 trainer/global_step‚10699 $ _timestamp‚1.788537463242592e+09 _runtime‚12846.778199037 _step‚876Þ:MãÏë+É ' lr-AdamW/pg1‚2.9114090038672216e-05 ' lr-AdamW/pg2‚2.9114090038672216e-05 trainer/global_step‚10699 $ _timestamp‚1.788537463242592e+09 _runtime‚12846.778199037 _step‚876ì”x^€4ì+/ _wandbruntime‚12846  _runtime‚12846Sí5 í+„P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚993794875142656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270882603 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885374632437458e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494378955 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223522696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64799488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223522696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚900695554248192 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133921796 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270882603 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚900695572251136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1006587467299840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚912945845042176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133921798 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚912944917067776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10699 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198721941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993794899082752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223522698 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12250272791040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12249362819584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚900695572251136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270856259 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494405301 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12204468601460 _step‚877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚29WX>(î+‰P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚912944917067776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10699 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198721941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993794899082752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223522698 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12250272791040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12249362819584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚900695572251136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270856259 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494405301 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12204468601460 _step‚877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚993794875142656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270882603 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885374632437458e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494378955 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223522696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64799488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223522696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚900695554248192 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133921796 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270882603 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚900695572251136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1006587467299840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚912945845042176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133921798 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521íhÙ±í4ï+/ _wandbruntime‚12847  _runtime‚12847Ôù1Ô'ð+ŽN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198739799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚913028585601536 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1006678867239424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚12847.913959465 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270881090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64805301 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871003870843904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133933861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993885124787200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12204677899734 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚900777173334016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198739162 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133933863 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223542858 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270881090 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12250484279808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223542856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270907460 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223542856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494450318 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270907460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚913027657613824 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494423946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494450318 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223542858 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1006678620822016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12793742452224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12250484279808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 trainer/global_step‚10699 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚913028585601536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12251394264576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993885100847104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494423946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12205585874946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883208548743638 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚900777191336960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883209473758210 _step‚878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64805936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12793519974912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0‘Â|~ñ+“N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270907460 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223542856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494450318 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270907460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚913027657613824 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494423946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494450318 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223542858 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1006678620822016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12793742452224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12250484279808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 trainer/global_step‚10699 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚913028585601536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12251394264576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993885100847104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494423946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12205585874946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883208548743638 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚900777191336960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883209473758210 _step‚878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor¤{Á¾›.on_train_batch_end/inactive_split.small_pool.allocated‚64805936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12793519974912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198739799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚913028585601536 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1006678867239424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚12847.913959465 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270881090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64805301 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871003870843904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133933861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993885124787200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12204677899734 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚900777173334016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198739162 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133933863 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223542858 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270881090 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12250484279808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223542856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637îj K4ò+/ _wandbruntime‚12847  _runtime‚12847¤?å"[ó+Õ /loss/FRt1/detection_loss‚3.2015058944622674 1loss/SRqqt3/detection_loss‚0.7478500552192007 1loss/SRqqt4/detection_loss‚1.8643179092651758 .loss/FRt1/assignment_loss‚2.42770724495252 /loss/FRt2/detection_loss‚2.3947363520223006 .loss/FBt4/detection_loss‚2.274922526401022 $ _timestamp‚1.788537464379808e+09 /loss/FRt3/assignment_loss‚0.664501008234526 0loss/FBt1/assignment_loss‚0.1823387476518595 1loss/FBt2/assignment_loss‚0.26923466103539734 /loss/FBt1/detection_loss‚2.8005745808283486 _step‚879 0loss/FRt2/assignment_loss‚3.0731238524119058 1loss/SRqqt4/assignment_loss‚1.291976351195421 1loss/SRqqt1/detection_loss‚2.1666415189251755 1loss/SRqqt2/detection_loss‚2.0892077004536986 -loss/FRt4/detection_loss‚2.24601209892167 &loss/total_loss‚38.866155232379384 1loss/FBt3/assignment_loss‚0.09164348380842978 trainer/global_step‚10699 _runtime‚12847.915255621 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2402679574443027 2loss/SRqqt3/assignment_loss‚0.7727904244447532 1loss/FBt4/assignment_loss‚0.20638390400022366 /loss/FBt2/detection_loss‚2.3735029300053916 /loss/FRt4/assignment_loss‚3.193957034746806 2loss/SRqqt1/assignment_loss‚1.2708551841929103 .loss/FRt3/detection_loss‚1.496675355457946 /loss/FBt3/detection_loss‚0.5254284562981304a´`ô+Ú /loss/FRt1/detection_loss‚3.2015058944622674 1loss/SRqqt3/detection_loss‚0.7478500552192007 1loss/SRqqt4/detection_loss‚1.8643179092651758 .loss/FRt1/assignment_loss‚2.42770724495252 /loss/FRt2/detection_loss‚2.3947363520223006 .loss/FBt4/detection_loss‚2.274922526401022 $ _timestamp‚1.788537464379808e+09 /loss/FRt3/assignment_loss‚0.664501008234526 0loss/FBt1/assignment_loss‚0.1823387476518595 1loss/FBt2/assignment_loss‚0.26923466103539734 /loss/FBt1/detection_loss‚2.8005745808283486 _step‚879 0loss/FRt2/assignment_loss‚3.0731238524119058 1loss/SRqqt4/assignment_loss‚1.291976351195421 1loss/SRqqt1/detection_loss‚2.1666415189251755 1loss/SRqqt2/detection_loss‚2.0892077004536986 -loss/FRt4/detection_loss‚2.24601209892167 &loss/total_loss‚38.866155232379384 1loss/FBt3/assignment_loss‚0.09164348380842978 trainer/global_step‚10699 _runtime‚12847.915255621 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2402679574443027 2loss/SRqqt3/assignment_loss‚0.7727904244447532 1loss/FBt4/assignment_loss‚0.20638390400022366 /loss/FBt2/detection_loss‚2.3735029300053916 /loss/FRt4/assignment_loss‚3.193957034746806 2loss/SRqqt1/assignment_loss‚1.2708551841929103 .loss/FRt3/detection_loss‚1.496675355457946 /loss/FBt3/detection_loss‚0.5254284562981304ïã¬J¼wõ+‚(:ì ÿÔëÔ¥¨òâ disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1837.671875 disk.rbd2.in‚0% proc.memory.availableMB‚ 1382.8125) disk./.usagePercent‚76.09139482879021" disk.nvme0n1.out‚ 146519.015625! proc.memory.rssMB‚ 2902.359375* proc.memory.percent‚17.714595794677734 cpu‚25.216329983686453 disk.nvme1n1.in‚ 850.26171875 network.sent‚ 2.8633216e+07% disk./.usageGB‚2721.3968467712402 disk.rbd0.out‚ 3.87109375 disk.sda1.in‚0! disk.nvme0n1.in‚ 7676.55078125 disk.md0.out‚ 1837.671875 disk.dm-0.in‚ 8526.8125 disk.rbd2.out‚ 3.87109375 disk.md0.in‚ 208.765625! disk.sdb.out‚1889.11669921875" disk.sdb2.out‚1889.11669921875! disk.sda.out‚1889.11669921875 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875! disk.nvme1n1.out‚ 63665.453125 network.recv‚ 4.278711e+06 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1889.11669921875 disk.sdb2.in‚ 75.59765625$ memory_percent‚91.55998229980469 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 210184.46875†ÿ1“ö+‚(:ˆ ÿÔëÔ›ÎÛá gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã ÎK“÷+‚(:ˆ ŽÕëÔã¨à gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úçÌø+‚(:ô ŽÕëÔÀÐÑà disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 disk.dm-0.in‚ 8526.8125 disk.sdb1.out‚0) disk./.usagePercent‚76.08378321323642 disk.md0p1.out‚ 1839.12890625( proc.memory.availableMB‚ 1382.3203125 network.sent‚ 2.8742953e+07 disk.rbd2.out‚ 3.87109375! disk.sdb2.out‚1890.6337890625 disk.md0p1.in‚ 208.765625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0.out‚ 1839.12890625 disk.nvme1n1.in‚ 850.26171875! disk.nvme1n1.out‚ 63702.984375 disk.sdb1.in‚0 disk.md0.in‚ 208.765625 disk.sdb.out‚1890.6337890625 disk.sda.out‚1890.6337890625 disk.rbd0.in‚0! proc.memory.rssMB‚ 2902.359375 network.recv‚ 4.285256e+06% disk./.usageGB‚2721.1246185302734! disk.nvme0n1.in‚ 7676.55078125$ disk.nvme0n1.out‚146674.60546875 disk.sda2.in‚ 133.16796875 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.87109375$ memory_percent‚91.56298637390137! disk.sda2.out‚1890.6337890625! disk.dm-0.out‚210377.58984375 disk.sda1.in‚0* proc.memory.percent‚17.714595794677734 cpu‚25.216126282142415 proc.cpu.threads‚17 ‚§(yù+‚(:î ÕëÔ©„åâ disk.md0.out‚ 1841.2265625! disk.dm-0.out‚210542.33984375! disk.sdb2.out‚1892.8037109375 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1841.2265625 disk.rbd0.in‚0 network.sent‚ 2.87471e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 75.59765625 proc.cpu.threads‚17 disk.md0.in‚ 208.765625 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.out‚ 63703.625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7676.55078125 disk.sdb2.in‚ 75.59765625 disk.sda.out‚1892.8037109375) proc.memory.availableMB‚ 1382.31640625* proc.memory.percent‚17.714595794677734 cpu‚25.215905149785687 network.recv‚ 4.290234e+06 disk.sdb1.in‚0 disk.dm-0.in‚ 8526.8125 disk.nvme1n1.in‚ 850.26171875$ disk.nvme0n1.out‚146838.71484375 disk.sda2.in‚ 133.16796875$ memory_percent‚91.56301021575928! proc.memory.rssMB‚ 2902.359375! disk.sda2.out‚1892.8037109375) disk./.usagePercent‚76.08435736688082% disk./.usageGB‚2721.1451530456543 disk.sdb1.out‚0 disk.sdb.out‚1892.8037109375 disk.rbd2.out‚ 3.87109375Ò°·”ú+‚(:‰ ÕëÔס€Ž gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yrwžû+‚(:“ ¬ÕëÔ·°à gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,†')sü+‚(:è ¬ÕëÔº²Ëà disk.sdb1.out‚0 disk.dm-0.in‚ 8526.8125 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sdb.out‚ 1894.3828125 disk.sda.out‚ 1894.3828125) disk./.usagePercent‚76.05855159775771! disk.nvme0n1.in‚ 7676.55078125 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.87109375) proc.memory.availableMB‚ 1382.32421875! proc.memory.rssMB‚ 2902.359375 network.sent‚ 2.8755492e+07 disk.md0.out‚ 1842.74609375$ memory_percent‚91.56296253204346$ disk./.usageGB‚2720.222213745117 disk.rbd2.out‚ 3.87109375 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875# disk.nvme1n1.out‚63704.63671875* proc.memory.percent‚17.714595794677734 proc.cpu.threads‚17 network.recv‚ 4.294348e+06 disk.rbd2.in‚0" disk.nvme0n1.out‚ 146989.578125 disk.sda2.out‚ 1894.3828125 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875! disk.dm-0.out‚210694.21484375 disk.sdb2.out‚ 1894.3828125 disk.sdb1.in‚0 disk.md0.in‚ 208.765625 disk.md0p1.out‚ 1842.74609375 disk.rbd0.in‚0 cpu‚25.215566791954625¹œ|è4ý+/ _wandbruntime‚12902  _runtime‚12902ÑH“Êþ+Ä trainer/global_step‚10749 $ _timestamp‚1.788537519197937e+09 _runtime‚12902.733479216 _step‚880 ' lr-AdamW/pg1‚2.7084430102946887e-05 ' lr-AdamW/pg2‚2.7084430102946887e-05´µÕÏÿ+É ' lr-AdamW/pg1‚2.7084430102946887e-05 ' lr-AdamW/pg2‚2.7084430102946887e-05 trainer/global_step‚10749 $ _timestamp‚1.788537519197937e+09 _runtime‚12902.733479216 _step‚880ðj 4€,/ _wandbruntime‚12902  _runtime‚12902-?\ (,„P dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12851444492288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874875732976560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65092601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12306360216064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12259437881564 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199617649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12306360216064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199616994 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚904776526542336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496658203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 272127505 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1011157628797440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚904776508539392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12260345856568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 272101161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874875750015920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134525048 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1011157380269568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚998306160365056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚887135170858124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917081958784000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚12902.734915572 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚904776526542336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272127505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917081958784000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12305450244608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚998306184305152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 224530696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65091948 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12305450244608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 224530698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885375191990323e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917082886758400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231Û£A³‚,‰P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 224530698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885375191990323e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917082886758400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12851444492288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874875732976560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65092601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12306360216064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12259437881564 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199617649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12306360216064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199616994 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚904776526542336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496658203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 272127505 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1011157628797440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚904776508539392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12260345856568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 272101161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874875750015920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134525048 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1011157380269568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚998306160365056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚887135170858124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917081958784000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚12902.734915572 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚904776526542336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272127505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917081958784000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12305450244608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚998306184305152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 224530696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65091948 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12305450244608ñqËOu4ƒ,/ _wandbruntime‚12903  _runtime‚12903Þªê'„,ŽN VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496703220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874954750571104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65098465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12852408448000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12306571704832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12852624633856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12307481689600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917164699330048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65097821 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272152362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224550858 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1011248794517504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12307481689600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272125992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12261463130054 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚904858145628160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚887216230740518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224550856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917165627317760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199634932 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370}ùQì'…,“N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65097821 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272152362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224550858 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1011248794517504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12307481689600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272125992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12261463130054 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚904858145628160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚887216230740518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224550856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917165627317760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199634932 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496703220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874954750571104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65098465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12852408448000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12306571704832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12852624633856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12307481689600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917164699330048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ò,úh!4†,/ _wandbruntime‚12903  _runtime‚12903ˆg£©`‡,Ú 1loss/SRqqt3/detection_loss‚1.0253348241970743 2loss/SRqqt3/assignment_loss‚1.0578791490725932 1loss/FBt1/assignment_loss‚0.07059287633940887 1loss/FBt3/assignment_loss‚0.11908273990539933 /loss/FRt3/detection_loss‚1.5863132297527045 trainer/global_step‚10749 1loss/FBt4/assignment_loss‚0.06694387014921817 1loss/SRqqt2/detection_loss‚1.9215330817597978 -loss/FBt3/detection_loss‚0.37545473575729 $loss/total_loss‚40.7445477114294 /loss/FRt4/assignment_loss‚3.439849801783292 /loss/FRt2/detection_loss‚2.4979049516882865 epoch‚8 _step‚883 /loss/FRt2/assignment_loss‚3.208815625420323 2loss/SRqqt4/assignment_loss‚1.3558856946312718 /loss/FRt1/detection_loss‚2.8750002880891166 1loss/SRqqt4/detection_loss‚2.1394192145930395 /loss/FBt4/detection_loss‚2.6348489197817715 0loss/FRt3/assignment_loss‚1.1299045765772462 2loss/SRqqt2/assignment_loss‚1.2338257444665788 .loss/FRt4/detection_loss‚2.157269755483798 /loss/FBt1/detection_loss‚2.2328989505767822 % _timestamp‚1.7885375201653047e+09 _runtime‚12903.700583469 0loss/FRt1/assignment_loss‚3.0124766707420347 1loss/SRqqt1/detection_loss‚2.4032274255385766 2loss/SRqqt1/assignment_loss‚1.2272293854218264 /loss/FBt2/detection_loss‚2.9691632177148546 2loss/FBt2/assignment_loss‚0.003692981987114763qùù(eˆ,ß 1loss/FBt3/assignment_loss‚0.11908273990539933 /loss/FRt3/detection_loss‚1.5863132297527045 trainer/global_step‚10749 1loss/FBt4/assignment_loss‚0.06694387014921817 1loss/SRqqt2/detection_loss‚1.9215330817597978 -loss/FBt3/detection_loss‚0.37545473575729 $loss/total_loss‚40.7445477114294 /loss/FRt4/assignment_loss‚3.439849801783292 /loss/FRt2/detection_loss‚2.4979049516882865 epoch‚8 _step‚883 /loss/FRt2/assignment_loss‚3.208815625420323 2loss/SRqqt4/assignment_loss‚1.3558856946312718 /loss/FRt1/detection_loss‚2.8750002880891166 1loss/SRqqt4/detection_loss‚2.1394192145930395 /loss/FBt4/detection_loss‚2.6348489197817715 0loss/FRt3/assignment_loss‚1.1299045765772462 2loss/SRqqt2/assignment_loss‚1.2338257444665788 .loss/FRt4/detection_loss‚2.157269755483798 /loss/FBt1/detection_loss‚2.2328989505767822 % _timestamp‚1.7885375201653047e+09 _runtime‚12903.700583469 0loss/FRt1/assignment_loss‚3.0124766707420347 1loss/SRqqt1/detection_loss‚2.4032274255385766 2loss/SRqqt1/assignment_loss‚1.2272293854218264 /loss/FBt2/detection_loss‚2.9691632177148546 2loss/FBt2/assignment_loss‚0.003692981987114763 1loss/SRqqt3/detection_loss‚1.0253348241970743 2loss/SRqqt3/assignment_loss‚1.0578791490725932 1loss/FBt1/assignment_loss‚0.07059287633940887óaö´”‰,‚(:‰ »ÕëÔس¬ß gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Õh}€Š,‚(:õ »ÕëÔîÇà network.recv‚ 4.298151e+06% disk./.usageGB‚2720.3465003967285 disk.md0p1.in‚ 208.765625 disk.sda1.out‚0$ memory_percent‚91.56322479248047 cpu‚25.21527836483696) disk./.usagePercent‚76.06202670455477 disk.sdb1.in‚0! disk.sda.out‚1895.86474609375 disk.sdb.in‚ 75.59765625! disk.sdb.out‚1895.86474609375& proc.memory.availableMB‚ 1382.28125! proc.memory.rssMB‚ 2902.359375" disk.sdb2.out‚1895.86474609375 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875 disk.sdb1.out‚0" disk.sda2.out‚1895.86474609375 network.sent‚ 2.8771525e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375 disk.rbd2.out‚ 3.87109375! disk.nvme0n1.in‚ 7676.55078125* proc.memory.percent‚17.714595794677734 disk.md0p1.out‚ 1844.17578125 disk.rbd2.in‚0$ disk.nvme0n1.out‚147164.80078125 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8526.8125 disk.dm-0.out‚ 210869.671875 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚63704.87109375 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.md0.out‚ 1844.17578125 proc.cpu.threads‚179îml‹,‚(:á ÊÕëÔ¸ÁÔà disk.sda1.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875 network.sent‚ 2.8867641e+07 network.recv‚ 4.304239e+06 disk.md0.out‚ 1846.20703125 disk.sda.out‚ 1897.96875* proc.memory.percent‚17.714595794677734 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.in‚ 75.59765625 disk.md0p1.out‚ 1846.20703125! proc.memory.rssMB‚ 2902.359375 disk.md0p1.in‚ 208.765625$ memory_percent‚91.56167507171631 proc.cpu.threads‚17$ disk./.usageGB‚2720.346508026123 disk.sda2.out‚ 1897.96875 disk.rbd0.out‚ 3.87109375 cpu‚25.214990079841787 disk.dm-0.in‚ 8526.8125 disk.sdb2.out‚ 1897.96875! disk.nvme0n1.in‚ 7676.55078125! disk.dm-0.out‚211003.00390625 disk.nvme1n1.in‚ 850.26171875" disk.nvme1n1.out‚ 63705.0390625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0) disk./.usagePercent‚76.06202691787583 disk.rbd2.out‚ 3.87109375$ disk.nvme0n1.out‚147297.96484375 disk.sdb.out‚ 1897.96875) proc.memory.availableMB‚ 1382.53515625aTÔ¨‘Œ,‚(:† ÊÕëÔÃÀÄà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Y»ß”,‚(:‰ ÙÕëÔìž“à gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¡Ý¡~Ž,‚(:ó ÙÕëÔúÌéà* proc.memory.percent‚17.714595794677734 disk.sda.in‚ 133.16796875 disk.sda.out‚1899.3193359375 disk.rbd2.in‚0! disk.sda2.out‚1899.3193359375! disk.dm-0.out‚211092.96484375 disk.md0p1.out‚ 1847.49609375 cpu‚25.214779273628828 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875# disk.nvme1n1.out‚63705.28515625% disk./.usageGB‚2720.3586235046387 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1847.49609375! disk.nvme0n1.in‚ 7682.55078125# disk.nvme0n1.out‚147387.6796875 disk.rbd2.out‚ 3.87109375$ memory_percent‚91.56017303466797 disk.sdb1.out‚0 disk.sda1.in‚0 network.sent‚ 2.8873789e+07 disk.dm-0.in‚ 8532.8125 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1899.3193359375 disk.rbd0.in‚0& proc.memory.availableMB‚ 1382.78125 network.recv‚ 4.308982e+06) disk./.usagePercent‚76.06236567172586! disk.sdb2.out‚1899.3193359375 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0! proc.memory.rssMB‚ 2902.359375EI~/4,/ _wandbruntime‚12957  _runtime‚12957ÙÔ ‰Ê,Ä trainer/global_step‚10799 $ _timestamp‚1.788537573482274e+09 _runtime‚12957.018003232 _step‚884 ' lr-AdamW/pg1‚2.5125378769447512e-05 ' lr-AdamW/pg2‚2.5125378769447512e-05[êÖÏ‘,É $ _timestamp‚1.788537573482274e+09 _runtime‚12957.018003232 _step‚884 ' lr-AdamW/pg1‚2.5125378769447512e-05 ' lr-AdamW/pg2‚2.5125378769447512e-05 trainer/global_step‚10799ô>úï‰4’,/ _wandbruntime‚12957  _runtime‚12957nì­à¨ “,†P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚908857462830592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65383615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225538698 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12315315136672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12361537669632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _runtime‚12957.019212918 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921219000500224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65384233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135128298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 200512531 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135128296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12910271062528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273346063 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1002817445587456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1015727502547968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273372407 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1015727740590080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚885 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12362447641088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12361537669632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 225538696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891142852854796 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚908857462830592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12362447641088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878826612703760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921219928474624r¥h?(”,‹P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 225538696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚908857480833536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200511911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚921219928474624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498911105 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12910056960512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12316223111676 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 498884759 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚921219000500224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚908857462830592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65383615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225538698 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12315315136672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12361537669632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _runtime‚12957.019212918 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921219000500224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65384233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135128298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 200512531 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135128296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12910271062528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273346063 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1002817445587456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1015727502547968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273372407 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1015727740590080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚885 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12362447641088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12361537669632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 225538696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891142852854796 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚908857462830592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12362447641088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878826612703760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921219928474624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273370894 % _timestamp‚1.7885375748220782e+09 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1015818888364544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878905630298304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 trainer/global_step‚10799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498956122 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921301741046272 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1002907695232000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚908939099919360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12317340385162 _runtime‚12958.360306177 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225558856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878905647337664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12911217072640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921302669033984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚908939099919360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200529782 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65389421 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚908939081916416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12362659129856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200530405 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1002907671291904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 225558856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12362659129856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921302669033984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚908939081916416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273397264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12363569114624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12363569114624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135140361 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65390042 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1015819143170560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12911447938560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498929750 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ’Oè—,•N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12363569114624 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135140361 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65390042 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1015819143170560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12911447938560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498929750 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225558858 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 273370894 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225558858 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273370894 % _timestamp‚1.7885375748220782e+09 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1015818888364544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878905630298304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 trainer/global_step‚10799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498956122 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921301741046272 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10029076ñŽ‡gý95232000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚908939099919360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12317340385162 _runtime‚12958.360306177 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225558856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878905647337664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12911217072640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921302669033984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚908939099919360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200529782 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65389421 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚908939081916416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12362659129856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200530405 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1002907671291904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 225558856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12362659129856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921302669033984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚908939081916416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273397264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12363569114624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚925356970190848 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201407150 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12418535066112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501137661 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚925356042216448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882777509470320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501164007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1007328730809856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚912938417121792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12417625094656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895148684822740 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226546696 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925356970190848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚912938435124736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12371192391780 trainer/global_step‚10849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12417625094656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885376330959516e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020297571020800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882777492430960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 274617309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12418535066112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274590965 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1007328754749952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚13016.632096977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚912938417121792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226546698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65675604 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020297823742976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226546696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274590965 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135731548 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12372100366784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925356042216448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚912938435124736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226546698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65676227 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12969068993024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 365502464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0®ÆAL¨,‹P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501137661 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚925356042216448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882777509470320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501164007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1007328730809856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚912938417121792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12417625094656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895148684822740 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226546696 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925356970190848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚912938435124736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12371192391780 trainer/global_step‚10849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12417625094656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885376330959516e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020297571020800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882777492430960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 274617309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12418535066112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274590965 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1007328754749952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚13016.632096977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚912938417121792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226546698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65675604 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020297823742976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226546696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274590965 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135731548 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12372100366784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925356042216448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚912938435124736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226546698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65676227 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12969068993024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peå‰Ü?Å ak‚ 365502464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135731546 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201407775 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274617309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12968840210944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚925356970190848 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201407150 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12418535066112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144ù!¯Ì@4©,/ _wandbruntime‚13017  _runtime‚13017”œÐr'ª,N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913020036207616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚925439710750208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12970244565504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10849 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882856527064864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274642166 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201425737 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65682124 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226566856 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135743613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12418746554880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201425125 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12970019991040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913020054210560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895229744705134 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274615796 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135743611 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12419656539648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501209024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 501182652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 501182652 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12419656539648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12418746554880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895228819690562 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925439710750208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913020054210560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1007418980454400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1007418956514304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020389225019904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882856510025504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020388976505344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 501182652 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12419656539648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12418746554880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895228819690562 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925439710750208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 365502464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913020054210560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1007418980454400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1007418956514304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020389225019904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882856510025504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020388976505344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882856527064864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274642166 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201425737 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65682124 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226566856 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135743613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12418746554880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201425125 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12970019991040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913020054210560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895229744705134 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274615796 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135743611 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12419656539648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501209024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 501182652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 EÞ“¹,‚(:ˆ ïÖëÔý«¢à gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ÙP sº,‚(:è ïÖëÔà’€á disk.sdb2.out‚1915.423828125 disk.sda.out‚1915.423828125 disk.md0p1.out‚ 1863.01171875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sdb1.in‚0$ disk./.usageGB‚2721.178596496582! disk.nvme0n1.in‚ 7682.55078125# disk.nvme0n1.out‚149125.4453125 disk.sda2.out‚1915.423828125" disk.nvme1n1.out‚ 63710.4765625$ memory_percent‚92.02022552490234* proc.memory.percent‚18.171024322509766 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.8125 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.dm-0.out‚ 212835.921875) disk./.usagePercent‚76.08529245976594 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625 disk.sda1.in‚0! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.91367e+07& proc.memory.availableMB‚ 1307.40625 disk.rbd0.out‚ 3.87109375 disk.md0.out‚ 1863.01171875 disk.rbd2.out‚ 3.87109375 disk.rbd0.in‚0 cpu‚25.22545270820675 network.recv‚ 4.357571e+06 proc.cpu.threads‚17 disk.sdb.out‚1915.423828125 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26171875)•”»,‚(:‰ þÖëÔ®©Öâ gpu.0.gpu‚46.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.031' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.50775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[u¼,‚(:ê þÖëÔ‚ž²ã disk.md0p1.out‚ 1864.3125 disk.sdb1.in‚0( disk./.usagePercent‚76.0914203206574" disk.sda2.out‚1916.77685546875 disk.rbd0.out‚ 3.87109375* proc.memory.percent‚18.171024322509766 network.recv‚ 4.361371e+06 disk.nvme0n1.out‚ 149479.625 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625" disk.sdb2.out‚1916.77685546875 disk.sda.in‚ 133.16796875$ memory_percent‚92.02022552490234 proc.cpu.threads‚17 network.sent‚ 2.9140138e+07 disk.sda2.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚63710.69140625 disk.sdb1.out‚0 cpu‚25.229791954069015 disk.sda1.out‚0! disk.nvme0n1.in‚ 7682.55078125 disk.nvme1n1.in‚ 850.26171875& proc.memory.availableMB‚ 1307.40625% disk./.usageGB‚2721.3977584838867 disk.rbd2.in‚0 disk.dm-0.in‚ 8532.8125 disk.rbd0.in‚0 disk.sda1.in‚0! disk.dm-0.out‚213190.31640625! disk.sda.out‚1916.77685546875 disk.md0p1.in‚ 208.765625! proc.memory.rssMB‚ 2977.140625 disk.rbd2.out‚ 3.87109375 disk.md0.out‚ 1864.3125! disk.sdb.out‚1916.77685546875ÿ_M¹4½,/ _wandbruntime‚13116  _runtime‚13116*Ils*¾,íT ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚930687475750400 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 275844528 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚66122855 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1012973087428096 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 275823817 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12431116368040 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚887776951649744 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚918209012311552 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503773957 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚66123078 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 227950142 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7885377332270782e+09 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚918209030314496 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 194441728 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503773957 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12478445435904 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202709244 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202709019 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1026003991872512 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 227950142 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚887776934610384 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12478445435904 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 136586164 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚900208050978424 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 227950140 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503794670 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0‘˜@x*¿,òT K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503773957 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12478445435904 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202709244 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202709019 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1026003991872512 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 227950142 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚887776934610384 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12478445435904 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 136586164 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚900208050978424 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 227950140 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503794670 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚930687475750400 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 275844528 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚66122855 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1012973087428096 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 275823817 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12431116368040 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚887776951649744 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚918209012311552 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503773957 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚66123078 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 227950142 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7885377332270782e+09 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚918209030314496 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 194441728 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570ü–[9‹4À,/ _wandbruntime‚13117  _runtime‚13117Wm.pxÁ,öR LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 275832631 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚226 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 503818960 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202724155 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚12432285646372 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1013029354981888 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚10880 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 227965616 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚66128203 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚13031223793664 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 503798247 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚887827377754928 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚918263291221504 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1013029331041792 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚900259663401300 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚918263273218560 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1026060554835456 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚930741525466112 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 136595728 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚12431604578924 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _step‚893 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12478935286784 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 _runtime‚13117.880730899 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 503798247 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1026060749277184 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 227965618 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚930742226508288 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚224 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚930742226508288 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 227965618 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12478935286784 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚918263291221504 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1013029331041792 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚900259663401300 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚918263273218560 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1026060554835456 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚930741525466112 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 136595728 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚12431604578924 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _step‚893 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12478935286784 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 _runtime‚13117.880730899 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 503798247 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1026060749277184 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 227965618 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚930742226508288 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚224 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚930742226508288 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 227965618 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12478935286784 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07626304030418396 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15461130440235138 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9883268475532532 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6495512127876282 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18905213475227356 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08445852994918823 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.49954959750175476 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9937856197357178 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.714110255241394 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5753471851348877 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_4_of_5‚1 particle/accuracy_4_of_7‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3925212025642395 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5492217540740967 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5565369725227356 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005319636780768633 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5074793100357056 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2053200900554657 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18292652070522308 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2879869341850281 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988309144973755 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.501286506652832 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47374746203422546 particle/accuracy_1_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18292652070522308 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989462673664093 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45911523699760437 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29316040873527527 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20618915557861328 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620977401733 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.005949804093688726 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24588370323181152 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5480064749717712 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6446428298950195 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33915525674819946 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15106600522994995 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24859757721424103 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2610047459602356 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4790496826171875 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1808369755744934 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906195998191833 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1710478663444519 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9202933311462402 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4653143584728241 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33944234251976013 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.4127082824707031 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39374998211860657 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4881688952445984 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6597222089767456 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919099807739258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017346741631627083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3823813498020172 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5400000214576721 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07321429252624512 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45911523699760437 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12511709332466125 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4753086566925049 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5026836395263672 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5852783918380737 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.025755492970347404 +jet/accuracy_3_of_5‚0.09037377685308456 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1288074404001236 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013907785527408123 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1881030797958374 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017920130863785744 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909287095069885 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4896205961704254 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_4_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1307651549577713 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4631599187850952 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5493420958518982 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4616151750087738 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3491923213005066 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1898401975631714 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9519776701927185 +jet/accuracy_2_of_5‚0.47932031750679016 jet/accuracy_2_of_7‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.294222891330719 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8526890277862549 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5733937621116638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5163359642028809 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08004231005907059 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4645504057407379 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7454676628112793 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5204623341560364 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920836091041565 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5764540433883667 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4620857834815979 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_1_of_1‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17430369555950165 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222238779068 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event´{c‘Q_proportion‚0 *jet/accuracy_1_of_1‚0.4681752920150757 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.44675928354263306 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5413658618927002 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39948520064353943 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4272927939891815 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 _runtime‚13134.506991416 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1330459862947464 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4789920747280121 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2497657686471939 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056737393140793 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04780476912856102 particle/accuracy_1_of_2‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3137873113155365 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010587337426841259 jet/accuracy_7_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2559523284435272 *jet/accuracy_2_of_6‚0.5705128312110901 particle/accuracy_3_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 epoch‚8 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.256939560174942 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6022160053253174 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20645171403884888 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5155038833618164 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5025709867477417 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47606706619262695 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883157014846802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17927780747413635 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4975534975528717 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1763620376586914 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4308779537677765 +jet/accuracy_3_of_4‚0.08245865255594254 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.578668475151062 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022167488932609558 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4937717914581299 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24694137275218964 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21889474987983704 +jet/accuracy_2_of_2‚0.12225453555583954 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.283071905374527 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5462962985038757 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 +particle/specificity‚0.9474291801452637 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924592971801758 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_2_of_2‚1 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3406573235988617 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37227192521095276 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11625967174768448 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4650516211986542 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.572941780090332 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5284482836723328 ,jet/accuracy_4_of_5‚0.016308244317770004 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47971415519714355 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.26620370149612427 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4721262454986572 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2418646365404129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4540286958217621 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9780018925666809 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8643130660057068 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8196272253990173 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5510994791984558 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18505269289016724 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43932458758354187 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920222163200378 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4734848737716675 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000003278255463 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4937102794647217 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0229032002389431 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,jet/accuracy_4_of_4‚0.005496545694768429 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_6‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19227536022663116 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24626433849334717 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907159805297852 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026082590222358704 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.4192039668560028 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42859357595443726 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16829952597618103 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/1FRt2SRqqt1FBt/event_purity‚0.0178160909563303 particle/accuracy_5_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 % _timestamp‚1.7885377509686353e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23427902162075043 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.189406156539917 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5067047476768494 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5427035689353943 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4294116795063019 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14942917227745056 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4584634006023407 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.39262521266937256 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10874666273593903 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.574999988079071 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3314814865589142 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026909254491329193 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚39.98490488843943 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 6validation_average_jet_accuracy‚0.4328882396221161 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4819485545158386 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.46926259994506836 particle/accuracy_2_of_4‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47372233867645264 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5607142448425293 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2740662395954132 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5333054661750793 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1594650149345398 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4965314567089081 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03280827775597572 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16954022645950317 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚894 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 jet/accuracy_3_of_7‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49763956665992737 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029402872547507286 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1389392614364624 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18459704518318176 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt2SRqqt0FBt/event_purity‚0.03498550131917 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893686175346375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.148853600025177 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38152652978897095 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08320426940917969 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4665708839893341 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3694940507411957 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5121212005615234 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/1FRt4SRqqt*FBt/event_purity‚0CZA ×.Ç,´¦ 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3137873113155365 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010587337426841259 jet/accuracy_7_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2559523284435272 *jet/accuracy_2_of_6‚0.5705128312110901 particle/accuracy_3_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 epoch‚8 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.256939560174942 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6022160053253174 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20645171403884888 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5155038833618164 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5025709867477417 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47606706619262695 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883157014846802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17927780747413635 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4975534975528717 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1763620376586914 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4308779537677765 +jet/accuracy_3_of_4‚0.08245865255594254 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.578668475151062 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022167488932609558 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4937717914581299 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24694137275218964 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21889474987983704 +jet/accuracy_2_of_2‚0.12225453555583954 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.283071905374527 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5462962985038757 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 +particle/specificity‚0.9474291801452637 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924592971801758 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_2_of_2‚1 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3406573235988617 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37227192521095276 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11625967174768448 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9881848096847534 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4650516211986542 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.572941780090332 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5284482836723328 ,jet/accuracy_4_of_5‚0.016308244317770004 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47971415519714355 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.26620370149612427 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4721262454986572 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2418646365404129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4540286958217621 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9780018925666809 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8643130660057068 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8196272253990173 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5510994791984558 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18505269289016724 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43932458758354187 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920222163200378 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4734848737716675 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000003278255463 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4937102794647217 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0229032002389431 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,jet/accuracy_4_of_4‚0.005496545694768429 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_6‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19227536022663116 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24626433849334717 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907159805297852 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026082590222358704 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.4192039668560028 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42859357595443726 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16829952597618103 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/1FRt2SRqqt1FBt/event_purity‚0.0178160909563303 particle/accuracy_5_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 % _timestamp‚1.7885377509686353e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23427902162075043 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.189406156539917 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5067047476768494 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & PurÎùÚœddity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5427035689353943 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4294116795063019 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14942917227745056 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4584634006023407 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.39262521266937256 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10874666273593903 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.574999988079071 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3314814865589142 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026909254491329193 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚39.98490488843943 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 6validation_average_jet_accuracy‚0.4328882396221161 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4819485545158386 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.46926259994506836 particle/accuracy_2_of_4‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47372233867645264 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5607142448425293 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2740662395954132 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5333054661750793 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1594650149345398 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4965314567089081 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03280827775597572 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16954022645950317 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚894 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 jet/accuracy_3_of_7‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49763956665992737 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029402872547507286 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1389392614364624 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18459704518318176 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7"Purity/*FRt2SRqqt0FBt/event_purity‚0.03498550131917 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893686175346375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.148853600025177 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38152652978897095 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08320426940917969 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4665708839893341 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3694940507411957 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5121212005615234 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206268787384 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2209891378879547 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 trainer/global_step‚10880 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15247748792171478 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24287526309490204 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.4361110329627991 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6424399018287659 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 jet/accuracy_6_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.992390513420105 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15580809116363525 particle/accuracy_1_of_3‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5034624338150024 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5043748617172241 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07626304030418396 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15461130440235138 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9883268475532532 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6495512127876282 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18905213475227356 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08445852994918823 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.49954959750175476 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9937856197357178 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.714110255241394 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5753471851348877 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_4_of_5‚1 particle/accuracy_4_of_7‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3925212025642395 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5492217540740967 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5565369725227356 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005319636780768633 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5074793100357056 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2053200900554657 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18292652070522308 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2879869341850281 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988309144973755 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.501286506652832 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47374746203422546 particle/accuracy_1_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18292652070522308 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989462673664093 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45911523699760437 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29316040873527527 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20618915557861328 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620977401733 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.005949804093688726 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24588370323181152 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5480064749717712 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6446428298950195 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33915525674819946 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15106600522994995 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24859757721424103 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2610047459602356 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4790496826171875 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1808369755744934 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906195998191833 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1710478663444519 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9202933311462402 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4653143584728241 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33944234251976013 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.4127082824707031 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39374998211860657 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4881688952445984 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6597222089767456 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919099807739258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017346741631627083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3823813498020172 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5400000214576721 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07321429252624512 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45911523699760437 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12511709332466125 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4753086566925049 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5026836395263672 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5852783918380737 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.025755492970347404 +jet/accuracy_3_of_5‚0.09037377685308456 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1288074404001236 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013907785527408123 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1881030797958374 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017920130863785744 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909287095069885 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4896205961704254 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_4_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1307651549577713 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4631599187850952 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5493420958518982 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4616151750087738 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3491923213005066 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1898401975631714 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9519776701927185 +jet/accuracy_2_of_5‚0.47932031750679016 jet/accuracy_2_of_7‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.294222891330719 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8526890277862549 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5733937621116638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5163359642028809 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08004231005907059 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4645504057407379 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7454676628112793 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5204623341560364 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920836091041565 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5764540433883667 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4620857834815979 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_1_of_1‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17430369555950165 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222238779068 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4681752920150757 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.44675928354263306 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5413658618927002 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39948520064353943 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4272927939891815 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 _runtime‚13134.506991416 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1330459862947464 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4789920747280121 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2497657686471939 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056737393140793 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04780476912856102 particle/accuracy_1_of_2‚1þêÆç”È,‚(:‰ œ×ëÔÞ¸Žá gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ y¯uÉ,‚(:ê œ×ëÔðÞ¾á disk.dm-0.out‚ 213584.90625$ disk./.usageGB‚2721.397243499756 disk.md0p1.in‚ 208.765625 disk.sdb.out‚1920.623046875$ proc.memory.availableMB‚1306.625 cpu‚25.237328185196642 proc.cpu.threads‚17 disk.rbd0.out‚ 3.87109375 disk.sda1.in‚0$ disk.nvme0n1.out‚149867.45703125 disk.md0.out‚ 1868.03515625 disk.dm-0.in‚ 8532.8125# disk.nvme1n1.out‚63717.44921875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.sda.in‚ 133.16796875 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.sda2.in‚ 133.16796875 network.sent‚ 2.9254307e+07 disk.sdb.in‚ 75.59765625 disk.sdb2.out‚1920.623046875 disk.sda.out‚1920.623046875 disk.md0p1.out‚ 1868.03515625 disk.sda2.out‚1920.623046875 disk.nvme1n1.in‚ 850.26171875$ memory_percent‚92.02499389648438! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.sdb2.in‚ 75.59765625 network.recv‚ 4.373354e+06) disk./.usagePercent‚76.09140592148556 disk.sdb1.out‚0 disk.md0.in‚ 208.765625½VÛ®”Ê,‚(:‰ «×ëÔêûÚà gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›âÔ>rË,‚(:ç «×ëÔœÌþá network.sent‚ 2.9377172e+07 disk.md0.out‚1869.375 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0# disk.nvme0n1.out‚150200.3515625$ proc.memory.availableMB‚1306.875 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375* proc.memory.percent‚18.171024322509766 disk.md0p1.out‚1869.375 disk.sdb1.in‚0! disk.nvme0n1.in‚ 7682.55078125 cpu‚25.237094019135988 network.recv‚ 4.379827e+06" disk.sda2.out‚1922.02685546875 disk.sdb.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625 disk.sda2.in‚ 133.16796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚92.02346801757812 proc.cpu.threads‚17$ disk./.usageGB‚2721.397186279297 disk.dm-0.in‚ 8532.8125! disk.sdb.out‚1922.02685546875" disk.sdb2.out‚1922.02685546875! disk.sda.out‚1922.02685546875 disk.nvme1n1.in‚ 850.26171875 disk.md0.in‚ 208.765625 disk.dm-0.out‚213917.9921875 disk.sdb2.in‚ 75.59765625! disk.nvme1n1.out‚ 63717.640625) disk./.usagePercent‚76.09140432157758‰ÿr"4Ì,/ _wandbruntime‚13156  _runtime‚13156jóžËÍ,Å % _timestamp‚1.7885377733908396e+09 _runtime‚13156.926381576 _step‚895 ' lr-AdamW/pg1‚2.1420720162764714e-05 ' lr-AdamW/pg2‚2.1420720162764714e-05 trainer/global_step‚10899ú4J×ÐÎ,Ê _step‚895 ' lr-AdamW/pg1‚2.1420720162764714e-05 ' lr-AdamW/pg2‚2.1420720162764714e-05 trainer/global_step‚10899 % _timestamp‚1.7885377733908396e+09 _runtime‚13156.926381576ÿ}í@§4Ï,/ _wandbruntime‚13156  _runtime‚13156™„VD Ð,†P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203236009 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228529686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 276419425 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12457844039953 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12505500084736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12505500084736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228529684 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12458752014957 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136937204 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028443599470592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889905216784752 _step‚896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136937202 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66299438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12504590113280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚902363060824705 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 276393081 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920437826557440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1028443864775680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚932942398667776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13058727549440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1015384895861248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚13156.927344931 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203236642 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015384871921152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚932943326642176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚932943326642176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13058968914432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504922765 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504949111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚920437808554496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66298807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228529686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚932942398667776 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276393081 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚10899êW×%(Ñ,‹P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1028443864775680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚932942398667776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13058727549440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1015384895861248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚13156.927344931 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203236642 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015384871921152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚932943326642176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚932943326642176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13058968914432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504922765 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504949111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚920437808554496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66298807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228529686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚932942398667776 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276393081 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚10899 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203236009 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228529686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 276419425 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12457844039953 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12505500084736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12505500084736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228529684 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12458752014957 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136937204 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028443599470592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889905216784752 _step‚896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136937202 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66299438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12504590113280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚902363060824705 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 276393081 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920437826557440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570€í é4Ò,/ _wandbruntime‚13157  _runtime‚13157uÔœ'Ó,N PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203253551 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015475121565696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136949269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12506621558272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933026067201536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12506621558272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228549844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265291776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12505711573504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12458961313231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276417912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241351680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1015475097625600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228549846 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66304896 trainer/global_step‚10899 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933026067201536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276444282 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028535242962432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889984251418656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 504967756 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902443195692527 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889984234379296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 504994128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504967756 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203254165 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136949267 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13059880045056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504994128 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028534977670656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276417912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66304284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933025139213824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12505711573504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228549844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276444282 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885377744571793e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228549846 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249‘67A»Ô,”N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136949267 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13059880045056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504994128 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028534977670656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276417912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66304284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933025139213824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12505711573504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228549844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276444282 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885377744571793e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228549846 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203253551 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015475121565696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136949269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12506621558272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933026067201536 N>DeviceStatsMonitor.on_train_batch_end/alloyfÔ_cated_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12506621558272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228549844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265291776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12505711573504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12458961313231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276417912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241351680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1015475097625600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228549846 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66304896 trainer/global_step‚10899 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933026067201536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276444282 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028535242962432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889984251418656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 504967756 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902443195692527 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889984234379296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 504994128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504967756 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203254165 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚04='ò4Õ,/ _wandbruntime‚13157  _runtime‚13157#¸ä^Ö,Ø 0loss/FRt4/assignment_loss‚2.5658318048868423 1loss/SRqqt2/detection_loss‚2.0301878903363204 epoch‚9 trainer/global_step‚10899 1loss/FBt4/assignment_loss‚0.24365133652995719 .loss/FRt3/detection_loss‚1.223860902885596 /loss/FRt4/detection_loss‚2.3786748498678207 _runtime‚13157.996116539 /loss/FRt2/assignment_loss‚3.539544916152954 2loss/SRqqt3/assignment_loss‚0.9138890493220012 0loss/FBt3/assignment_loss‚0.0518361176722323 0loss/SRqqt1/detection_loss‚2.039005024092538 0loss/SRqqt3/detection_loss‚0.679577630523004 %loss/total_loss‚37.70245852784397 $ _timestamp‚1.788537774460587e+09 1loss/FBt1/assignment_loss‚0.33056428025010975 1loss/SRqqt4/detection_loss‚2.0777852460741997 2loss/SRqqt1/assignment_loss‚1.3236933551196541 /loss/FRt2/detection_loss‚2.4769483131426386 /loss/FBt1/detection_loss‚2.1433406710624694 .loss/FBt2/detection_loss‚2.016012791792552 /loss/FBt4/detection_loss‚1.8764426239899226 0loss/FBt3/detection_loss‚0.49419916642065614 0loss/FRt1/assignment_loss‚2.0949030232429506 2loss/SRqqt4/assignment_loss‚1.4248346430249512 /loss/FRt1/detection_loss‚3.1262311899662016 _step‚898 0loss/FRt3/assignment_loss‚1.0723548714319866 1loss/SRqqt2/assignment_loss‚1.480618913733476 1loss/FBt2/assignment_loss‚0.09846991632293793ÀäÛc×,Ý trainer/global_step‚10899 1loss/FBt4/assignment_loss‚0.24365133652995719 .loss/FRt3/detection_loss‚1.223860902885596 /loss/FRt4/detection_loss‚2.3786748498678207 _runtime‚13157.996116539 /loss/FRt2/assignment_loss‚3.539544916152954 2loss/SRqqt3/assignment_loss‚0.9138890493220012 0loss/FBt3/assignment_loss‚0.0518361176722323 0loss/SRqqt1/detection_loss‚2.039005024092538 0loss/SRqqt3/detection_loss‚0.679577630523004 %loss/total_loss‚37.70245852784397 $ _timestamp‚1.788537774460587e+09 1loss/FBt1/assignment_loss‚0.33056428025010975 1loss/SRqqt4/detection_loss‚2.0777852460741997 2loss/SRqqt1/assignment_loss‚1.3236933551196541 /loss/FRt2/detection_loss‚2.4769483131426386 /loss/FBt1/detection_loss‚2.1433406710624694 .loss/FBt2/detection_loss‚2.016012791792552 /loss/FBt4/detection_loss‚1.8764426239899226 0loss/FBt3/detection_loss‚0.49419916642065614 0loss/FRt1/assignment_loss‚2.0949030232429506 2loss/SRqqt4/assignment_loss‚1.4248346430249512 /loss/FRt1/detection_loss‚3.1262311899662016 _step‚898 0loss/FRt3/assignment_loss‚1.0723548714319866 1loss/SRqqt2/assignment_loss‚1.480618913733476 1loss/FBt2/assignment_loss‚0.09846991632293793 0loss/FRt4/assignment_loss‚2.5658318048868423 1loss/SRqqt2/detection_loss‚2.0301878903363204 epoch‚9‚ë(üò”Ø,‚(:‰ º×ëÔÀ²ìá gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.735' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.68375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýîH@~Ù,‚(:ó º×ëÔÁ©¶â disk.dm-0.in‚ 8532.8125 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.sda.out‚1923.5693359375! disk.nvme1n1.out‚ 63717.828125 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1870.84765625$ memory_percent‚92.02253818511963 disk.rbd0.in‚0) disk./.usagePercent‚76.08379099945526 disk.nvme1n1.in‚ 850.26171875$ disk.nvme0n1.out‚150357.78515625 disk.sda2.in‚ 133.16796875 disk.sda.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.23684169570661 network.recv‚ 4.383623e+06 disk.sda1.in‚0 disk.sdb2.in‚ 75.59765625 network.sent‚ 2.9395636e+07 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375 disk.md0.out‚ 1870.84765625) proc.memory.availableMB‚ 1307.02734375$ disk./.usageGB‚2721.124897003174! disk.nvme0n1.in‚ 7682.55078125! disk.sda2.out‚1923.5693359375 disk.sdb.out‚1923.5693359375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚214075.61328125! disk.sdb2.out‚1923.5693359375’­±“Ú,‚(:ˆ É×ëÔí£ðß gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¼•1Û,‚(:ô É×ëÔªÂêà disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 3.87109375 network.sent‚ 2.949293e+07 disk.nvme1n1.in‚ 850.26171875! disk.nvme0n1.in‚ 7682.55078125$ memory_percent‚92.02253818511963! disk.sdb.out‚1925.42431640625" disk.sdb2.out‚1925.42431640625" disk.nvme1n1.out‚ 63718.4140625 proc.cpu.threads‚17% disk./.usageGB‚2721.1454544067383 disk.dm-0.out‚ 214328.328125 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375) disk./.usagePercent‚76.08436579306287 disk.dm-0.in‚ 8532.8125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625! disk.sda.out‚1925.42431640625 disk.md0p1.out‚ 1872.6484375# disk.nvme0n1.out‚150609.9140625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 cpu‚25.236616924394983 network.recv‚ 4.390675e+06 disk.md0.in‚ 208.765625 disk.md0.out‚ 1872.6484375 disk.sdb.in‚ 75.59765625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1307.02734375" disk.sda2.out‚1925.42431640625ÞÞM¹’Ü,‚(:‡ Ø×ëÔÄÈ«à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.4825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cûÓ{Ý,‚(:ð Ø×ëԻܛá) disk./.usagePercent‚76.08791897536943! disk.nvme0n1.out‚ 150811.90625 disk.sda2.in‚ 133.16796875 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb.in‚ 75.59765625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0$ memory_percent‚92.02408790588379! disk.sda2.out‚1927.0732421875 disk.dm-0.in‚ 8532.8125 network.recv‚ 4.394911e+06$ disk./.usageGB‚2721.272533416748 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625! disk.sdb2.out‚1927.0732421875 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1874.2421875* proc.memory.percent‚18.171024322509766 disk.sdb1.in‚0 disk.md0.out‚ 1874.2421875! disk.dm-0.out‚214530.49609375 disk.sda.in‚ 133.16796875# disk.nvme1n1.out‚63718.58984375 proc.cpu.threads‚17 network.sent‚ 2.9498357e+07! disk.nvme0n1.in‚ 7682.55078125 disk.sdb.out‚1927.0732421875 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 3.87109375( proc.memory.availableMB‚ 1306.7734375 disk.rbd2.out‚ 3.87109375 disk.sda.out‚1927.0732421875! proc.memory.rssMB‚ 2977.140625 cpu‚25.236384826949614)Â’Þ,‚(:‡ ç×ëÔŠ¡œà gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æIè•}ß,‚(:ò ç×ëÔ‰´Âà disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 208.765625! disk.dm-0.out‚214666.86328125) disk./.usagePercent‚76.06203363748935 disk.rbd0.out‚ 3.88671875 disk.dm-0.in‚ 8532.8125* proc.memory.percent‚18.171024322509766 network.recv‚ 4.398767e+06 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0 disk.sda1.out‚0! disk.sda2.out‚1928.6669921875! disk.sdb2.out‚1928.6669921875$ disk.nvme0n1.out‚150947.81640625 disk.sdb2.in‚ 75.59765625 disk.sdb1.in‚0! proc.memory.rssMB‚ 2977.140625 disk.sda.out‚1928.6669921875 disk.rbd2.out‚ 3.9453125 cpu‚25.23615753251023 proc.cpu.threads‚17 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1928.6669921875 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7682.55078125$ memory_percent‚92.02253818511963) proc.memory.availableMB‚ 1307.02734375 disk.md0.out‚ 1875.76953125 network.sent‚ 2.9503867e+07! disk.nvme1n1.out‚ 63719.046875 disk.sda2.in‚ 133.16796875$ disk./.usageGB‚2720.346748352051 disk.md0p1.out‚ 1875.76953125U¿¸Þ4à,/ _wandbruntime‚13215  _runtime‚13215[à1gËá,Å ' lr-AdamW/pg1‚1.9675884934882253e-05 ' lr-AdamW/pg2‚1.9675884934882253e-05 trainer/global_step‚10949 % _timestamp‚1.7885378314848635e+09 _runtime‚13215.020306765 _step‚899êüËÐâ,Ê trainer/global_step‚10949 % _timestamp‚1.7885378314848635e+09 _runtime‚13215.020306765 _step‚899 ' lr-AdamW/pg1‚1.9675884934882253e-05 ' lr-AdamW/pg2‚1.9675884934882253e-05ƒ¡­Š[4ã,/ _wandbruntime‚13215  _runtime‚13215¬vœø (ä,…P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019896181083648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚924518762845696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885378314858334e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚593 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937080368358400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277637983 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204122771 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204122176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229537686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 229537684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277664327 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13117039156736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13117270035968 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137540452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚13215.02143573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚906369817807013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚924518780848640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137540454 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1033013196300288 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12514629270065 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937079440384000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277637983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66582317 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937079440384000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _step‚900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346Ânáà å,ŠP 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚10949 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937080368358400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277664327 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937079440384000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _step‚900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019896181083648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚924518762845696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885378314858334e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚593 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937080368358400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277637983 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204122771 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204122176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229537686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 229537684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277664327 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13117039156736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13117270035968 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137540452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚13215.02143573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚906369817807013 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚924518780848640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137540454 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1033013196300288 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12514629270065 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937079440384000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277637983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66582317 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 229557846 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1033104839525376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12562708983296 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019986406788096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885378324318116e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277662814 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚901 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893935131145856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12561798998528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13118195579904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 229557846 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 507220658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277689184 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1033104578427904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 229557844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937162180930048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚924600399934464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1019986382848000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893935114106496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507247030 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204139841 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514838568339 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚924600399934464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507247030cþ°K'è,•N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚13215.967981588 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204139841 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514838568339 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚924600399934464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507247030 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 229557846 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1033104839525376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12562708983296 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019986406788096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885378324318116e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277662814 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚901 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893935131145856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12561798998528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13118195579904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 229557846 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 507220658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277689184 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1033104578427904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 229557844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937162180930048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚924600399934464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1019986382848000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893935114106496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507247030 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66866652 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205009744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12569598550169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚928599735139840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/reque!{ %5sted_bytes.all.freed‚910376574789321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 230545684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897806976239152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278882885 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138143702 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66866042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12617674934784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278909229 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941216482100224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 230545686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚10999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _runtime‚13268.835441662 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 230545684 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚928599735139840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13175229080064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278882885 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910377499803685 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024407466306048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897806993278512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885378852995152e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205010356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1­ø"Õ(÷,‹P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 230545684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897806976239152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278882885 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138143702 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66866042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12617674934784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278909229 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941216482100224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 230545686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚10999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _runtime‚13268.835441662 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 230545684 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚928599735139840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13175229080064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278882885 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910377499803685 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024407466306048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897806993278512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885378852995152e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205010356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚941217410074624 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1037582932556800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278909229 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1037582671446016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 230545686 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12617674934784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66866652 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205009744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12569598550169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚928599735139840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚910376574789321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ˆß³ìõ4ø,/ _wandbruntime‚13270  _runtime‚13270¹”™'ù,N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 230565846 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 509473560 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 230565844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12617886423552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278934086 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12618796408320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024497668070400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278907716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66871674 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205028074 % _timestamp‚1.7885378865948336e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12617886423552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230565846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205027441 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12570715823447 _runtime‚13270.132247235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _step‚905 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚928681354225664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941300150633984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚941299222646272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 230565844 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024497692010496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚928681354225664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509499932 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚928681336222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037674307817984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897886010873056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278934086 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509499932 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897885993833696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12618796408320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024497668070400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278907716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66871674 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205028074 % _timestamp‚1.7885378865948336e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12617886423552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230565846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205027441 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12570715823447 _runtime‚13270.132247235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _step‚905 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚928681354225664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941300150633984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚941299222646272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 230565844 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024497692010496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚928681354225664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509499932 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚928681336222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037674307817984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897886010873056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278934086 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509499932 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897885993833696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280154131 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138746952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 231553684 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511707817 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280127787 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1028918751528448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028918727588352 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280154131 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231553686 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚914384256785993 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚945354451790848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280127787 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042152575836160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12672852388352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12673762359808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚945354451790848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚11049 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231553686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901757855966352 _runtime‚13327.427460467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚932680689431040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12625475805277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚945353523816448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205898976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511681471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12673762359808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚932680689431040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914383331771629 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚945353523816448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205898367 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901757873005712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12626383780281 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚932680671428096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042152325211136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚932680671428096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13233824307712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12672852388352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231553684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138746954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚908ko€^‹-‹P RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205898976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511681471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12673762359808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚932680689431040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914383331771629 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚945353523816448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205898367 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901757873005712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12626383780281 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚932680671428096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042152325211136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚932680671428096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13233824307712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12672852388352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231553684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138746954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67151415 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280154131 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138746952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 231553684 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511707817 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280127787 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1028918751528448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028918727588352 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280154131 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231553686 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚914384256785993 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚945354451790848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280127787 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042152575836160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsN ³ Monitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12672852388352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12673762359808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚945354451790848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚11049 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231553686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901757855966352 _runtime‚13327.427460467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚932680689431040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12625475805277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚945353523816448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760Œ#on4Œ-/ _wandbruntime‚13328  _runtime‚13328b|'-N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231573846 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12673973848576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13234992640512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511752834 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11049 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205916735 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 231573844 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205916091 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280178988 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280152618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚932762308516864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12626593078555 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚914464391654023 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 231573846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13234755483136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚932762290513920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚932762290513920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67157074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511726462 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914463466639451 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚13328.526667807 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885379449891858e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚909 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511752834 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1029008953292800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042243708775936 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280152618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚945437192350208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚945436264362496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2×ÜA'Ž-•N F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚909 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511752834 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1029008953292800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042243708775936 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280152618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚945437192350208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚945436264362496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231573846 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12673973848576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13234992640512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511752834 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11049 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205916735 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 231573844 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205916091 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280178988 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280152618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚932762308516864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12626593078555 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚914464391654023 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 231573846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13234755483136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚932762290513920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚932762290513920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67157074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511726462 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914463466639451 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚13328.526667807 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885379449891858e+09×}\„4-/ _wandbruntime‚13328  _runtime‚13328FaÇŸW-Ñ 1loss/SRqqt1/detection_loss‚2.1689968810927485 trainer/global_step‚11049 0loss/FBt1/assignment_loss‚0.4180304397189805 0loss/SRqqt4/detection_loss‚2.072234338947705 2loss/SRqqt1/assignment_loss‚1.6493790332348115 .loss/FRt1/detection_loss‚3.112847646077474 .loss/FRt4/detection_loss‚2.265317085718866 /loss/FBt3/detection_loss‚0.5717949365543855 1loss/FBt3/assignment_loss‚0.04985473503336214 .loss/FRt2/detection_loss‚2.711288886955554 /loss/FRt3/detection_loss‚1.3716145962056132 % _timestamp‚1.7885379449927568e+09 0loss/FRt3/assignment_loss‚1.3827834485038635 2loss/SRqqt4/assignment_loss‚1.1279659269466287 %loss/total_loss‚39.41534450999356 .loss/FRt1/assignment_loss‚2.69795665393273 /loss/FRt2/assignment_loss‚2.739583646809613 /loss/FRt4/assignment_loss‚2.524521281552869 2loss/SRqqt3/assignment_loss‚0.8281001884774689 1loss/SRqqt3/detection_loss‚0.8419706593891702 /loss/FBt2/detection_loss‚2.0709398022064796 2loss/SRqqt2/assignment_loss‚1.5236387861271699 /loss/FBt4/assignment_loss‚0.273403030587275 1loss/SRqqt2/detection_loss‚2.0569881896177926 .loss/FBt1/detection_loss‚2.220932510164049 _runtime‚13328.528192074 _step‚910 0loss/FBt2/assignment_loss‚0.2178025546779206 .loss/FBt4/detection_loss‚2.517399251461029 epoch‚9ìKó\‘-Ö /loss/FBt2/detection_loss‚2.0709398022064796 2loss/SRqqt2/assignment_loss‚1.5236387861271699 /loss/FBt4/assignment_loss‚0.273403030587275 1loss/SRqqt2/detection_loss‚2.0569881896177926 .loss/FBt1/detection_loss‚2.220932510164049 _runtime‚13328.528192074 _step‚910 0loss/FBt2/assignment_loss‚0.2178025546779206 .loss/FBt4/detection_loss‚2.517399251461029 epoch‚9 1loss/SRqqt1/detection_loss‚2.1689968810927485 trainer/global_step‚11049 0loss/FBt1/assignment_loss‚0.4180304397189805 0loss/SRqqt4/detection_loss‚2.072234338947705 2loss/SRqqt1/assignment_loss‚1.6493790332348115 .loss/FRt1/detection_loss‚3.112847646077474 .loss/FRt4/detection_loss‚2.265317085718866 /loss/FBt3/detection_loss‚0.5717949365543855 1loss/FBt3/assignment_loss‚0.04985473503336214 .loss/FRt2/detection_loss‚2.711288886955554 /loss/FRt3/detection_loss‚1.3716145962056132 % _timestamp‚1.7885379449927568e+09 0loss/FRt3/assignment_loss‚1.3827834485038635 2loss/SRqqt4/assignment_loss‚1.1279659269466287 %loss/total_loss‚39.41534450999356 .loss/FRt1/assignment_loss‚2.69795665393273 /loss/FRt2/assignment_loss‚2.739583646809613 /loss/FRt4/assignment_loss‚2.524521281552869 2loss/SRqqt3/assignment_loss‚0.8281001884774689 1loss/SRqqt3/detection_loss‚0.8419706593891702Ž,Þ•'”’-‚(:‰ ßØëÔìØá gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hׄ“-‚(:ù ßØëÔ‚Ìâ disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.07742432563238 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚ 4.16259765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 4.150390625 proc.cpu.threads‚17$ disk.nvme0n1.out‚151758.80078125 disk.md0.in‚ 208.765625! disk.dm-0.out‚215888.71484375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.23425255224639! disk.nvme0n1.in‚ 7682.55078125 disk.md0.out‚ 1888.2890625 disk.dm-0.in‚ 8532.8125 disk.sda.in‚ 133.16796875! disk.sda.out‚1941.64794921875 network.sent‚ 2.9765362e+07% disk./.usageGB‚2720.8971939086914! disk.sdb.out‚1941.64794921875 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.out‚ 1888.2890625$ memory_percent‚92.02451705932617' proc.memory.availableMB‚ 1306.703125 network.recv‚ 4.438396e+06 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0" disk.sdb2.out‚1941.64794921875" disk.nvme1n1.out‚ 64129.9140625 disk.sdb1.in‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1941.64794921875î3™9“”-‚(:ˆ îØëÔåäÔà gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Öü³‚•-‚(:÷ îØëÔ߯Ûá disk.sdb1.out‚0 disk.sdb.out‚1943.3818359375* proc.memory.percent‚18.171024322509766 disk.rbd0.in‚0 disk.sdb.in‚ 75.59765625# disk.nvme1n1.out‚64263.08203125# disk.nvme0n1.out‚151862.4765625 disk.dm-0.in‚ 8532.8125 cpu‚25.23404946372289 network.recv‚ 4.44429e+06 disk.sda1.in‚0! disk.sda2.out‚1943.3818359375 disk.md0.out‚ 1889.95703125 disk.sda.out‚1943.3818359375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7682.55078125! disk.dm-0.out‚216125.55859375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1889.95703125! disk.sdb2.out‚1943.3818359375( proc.memory.availableMB‚ 1306.8984375! proc.memory.rssMB‚ 2977.140625 network.sent‚ 2.9862386e+07 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚ 4.16259765625 disk.rbd2.out‚ 4.150390625 disk.md0.in‚ 208.765625$ memory_percent‚92.02332496643066) disk./.usagePercent‚76.06979948417259% disk./.usageGB‚2720.6244926452637 disk.sda.in‚ 133.16796875 disk.sdb1.in‚0]oOP“–-‚(:ˆ ýØëÔ¿î´à gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.874' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.9685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä7x‚—-‚(:÷ ýØëÔøæžá disk.sdb2.in‚ 75.59765625! disk.sdb2.out‚1944.7705078125 disk.rbd2.in‚0 disk.sda.in‚ 133.16796875 disk.rbd0.out‚ 4.16259765625 disk.sdb1.out‚0$ disk.nvme0n1.out‚151966.69921875* proc.memory.percent‚18.171024322509766! disk.dm-0.out‚216267.38671875 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1944.7705078125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1891.27734375! disk.sda2.out‚1944.7705078125 proc.cpu.threads‚17 network.sent‚ 2.9866324e+07 disk.sda.out‚1944.7705078125 disk.nvme1n1.in‚ 850.26171875 disk.rbd0.in‚0 disk.md0.in‚ 208.765625) proc.memory.availableMB‚ 1306.89453125 cpu‚25.233826384435517 disk.rbd2.out‚ 4.150390625 network.recv‚ 4.448768e+06 disk.dm-0.in‚ 8532.8125 disk.md0p1.out‚ 1891.27734375 disk.sda1.out‚0! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.08275415242251 disk.md0p1.in‚ 208.765625! disk.nvme0n1.in‚ 7682.55078125$ memory_percent‚92.02334880828857 disk.nvme1n1.out‚ 64300.6875% disk./.usageGB‚2721.0878143310547\”5%“˜-‚(:ˆ ŒÙëԊΨà gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”(#x™-‚(:í ŒÙëÔâÝà disk.nvme1n1.in‚ 850.26171875 proc.cpu.threads‚17) disk./.usagePercent‚76.08439981777262 disk.rbd0.in‚0 disk.sda1.in‚0 disk.md0.in‚ 208.765625! disk.sdb2.out‚1946.3173828125" disk.nvme1n1.out‚ 64300.8359375 disk.sda1.out‚0# disk.nvme0n1.out‚152130.8203125 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8532.8125 disk.dm-0.out‚ 216431.65625$ memory_percent‚92.02489852905273' proc.memory.availableMB‚ 1306.640625 network.recv‚ 4.453035e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625! disk.sda2.out‚1946.3173828125 disk.sda.in‚ 133.16796875 cpu‚25.233604016226415 disk.rbd0.out‚ 4.16259765625! disk.nvme0n1.in‚ 7682.55078125 disk.md0.out‚ 1892.765625 disk.sdb.out‚1946.3173828125! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766 disk.md0p1.in‚ 208.765625 network.sent‚ 2.98726e+07 disk.sdb1.out‚0 disk.sdb.in‚ 75.59765625 disk.sda.out‚1946.3173828125$ disk./.usageGB‚2721.146671295166 disk.sdb2.in‚ 75.59765625 disk.md0p1.out‚ 1892.765625 disk.sdb1.in‚0ÄØÙt4š-/ _wandbruntime‚13387  _runtime‚13387ᇰIË›-Å _runtime‚13387.781768381 _step‚911 ' lr-AdamW/pg1‚1.4875673671986763e-05 ' lr-AdamW/pg2‚1.4875673671986763e-05 trainer/global_step‚11099 % _timestamp‚1.7885380042460666e+09²ÁÛmÐœ-Ê _runtime‚13387.781768381 _step‚911 ' lr-AdamW/pg1‚1.4875673671986763e-05 ' lr-AdamW/pg2‚1.4875673671986763e-05 trainer/global_step‚11099 % _timestamp‚1.7885380042460666e+09þòo4-/ _wandbruntime‚13387  _runtime‚13387 ý#ù%ž-†P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232561684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232561684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚949491493507072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281399033 QCDeviceStatsMonitor.on_train_batc¤f Tç%h_start/allocation.small_pool.freed‚ 281372689 % _timestamp‚1.7885380042476888e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13291991265280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚11099 _step‚912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12728939813376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 232561686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206787665 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚949490565532672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚949491493507072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206787046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12728939813376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1033430012810752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚936761643722240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚936761643722240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13292228435968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67437461 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12681353060385 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1046722004076032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚949490565532672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281372689 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682261035389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185~æñÈ(Ÿ-‹P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67437461 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12681353060385 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1046722004076032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚949490565532672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281372689 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682261035389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232561684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232561684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚949491493507072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281399033 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281372689 % _timestamp‚1.7885380042476888e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13291991265280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚11099 _step‚912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12728939813376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 232561686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206787665 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚949490565532672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚949491493507072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206787046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12728939813376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1033430012810752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚936761643722240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚936761643722240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13292228435968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ì*!>4 -/ _wandbruntime‚13388  _runtime‚13388hƒ•''¡-N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 281397520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 281397520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 232581846 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514005736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 514005736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513979364 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚949574234066432 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1046813661151232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚13388.997454327 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11099 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12730971258368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 232581846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206804697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67442430 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1033520238515200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚949573306078720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13293398695936 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 232581844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139362267 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513979364 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1033520262455296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67443028 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚913 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206805297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚949574234066432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12730061273600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281423890 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1046813421025280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13293182510080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918470223621759 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885380054592416e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12730061273600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 232581844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514005736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 514005736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513979364 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚949574234066432 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1046813661151232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚13388.997454327 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11099 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12730971258368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 232581846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206804697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67442430 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1033520238515200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚949573306078720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13293398695936 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 232581844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139362267 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513979364 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1033520262455296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67443028 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚913 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206805297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚949574234066432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12730061273600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281423890 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1046813421025280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13293182510080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918470223621759 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885380054592416e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12730061273600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185‘0‹ÎS4£-/ _wandbruntime‚13389  _runtime‚13389 J`¤-Ú .loss/FRt3/detection_loss‚1.282341490170973 1loss/SRqqt2/detection_loss‚2.1440215706825256 1loss/FBt1/assignment_loss‚0.06895431755785669 /loss/FRt1/detection_loss‚2.7576962328619428 1loss/SRqqt3/detection_loss‚0.9824125171790636 epoch‚9 .loss/FRt2/detection_loss‚2.402291930695167 .loss/FRt1/assignment_loss‚2.13190397951338 2loss/SRqqt4/assignment_loss‚1.0361251043421882 1loss/FBt4/assignment_loss‚0.08123253541599826 /loss/FRt4/assignment_loss‚2.559800308280521 1loss/SRqqt4/detection_loss‚2.0325890902962005 /loss/FBt3/detection_loss‚0.4266206045052968 _runtime‚13389.002143179 /loss/FBt2/detection_loss‚2.2368234544992447 /loss/FBt4/detection_loss‚2.4003866420072666 % _timestamp‚1.7885380054633942e+09 _step‚914 2loss/SRqqt3/assignment_loss‚1.0925588185335835 /loss/FRt4/detection_loss‚2.1783117165168124 %loss/total_loss‚38.15589284392685 trainer/global_step‚11099 0loss/FRt2/assignment_loss‚2.5797390177332122 1loss/SRqqt1/assignment_loss‚1.307540354820398 1loss/SRqqt1/detection_loss‚2.7697053528749027 /loss/FBt1/detection_loss‚2.6387820641199746 0loss/FRt3/assignment_loss‚1.2063357593382107 2loss/SRqqt2/assignment_loss‚1.6677645201395666 1loss/FBt2/assignment_loss‚0.14631412480838435 2loss/FBt3/assignment_loss‚0.025641337034176103©M3g˯-Å ' lr-AdamW/pg1‚1.3421479573252245e-05 ' lr-AdamW/pg2‚1.3421479573252245e-05 trainer/global_step‚11149 % _timestamp‚1.7885380646512973e+09 _runtime‚13448.186972324 _step‚915†dÊxа-Ê ' lr-AdamW/pg2‚1.3421479573252245e-05 trainer/global_step‚11149 % _timestamp‚1.7885380646512973e+09 _runtime‚13448.186972324 _step‚915 ' lr-AdamW/pg1‚1.3421479573252245e-05“ƒ¬JÛ4±-/ _wandbruntime‚13448  _runtime‚13448f€hŽ (²-†P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12737230315493 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139953454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12785027238400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚940842580010496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1037941298033152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1051291684740608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 233569686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚940842598013440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚940842598013440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207677577 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13350619683840 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 516187275 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282643935 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 516213621 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12738138290497 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚953628535223296 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 233569686 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚953627607248896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 233569684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282617591 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615•v«Å³-‹P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12785027238400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67723493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚909659615420752 % _timestamp‚1.7885380646529083e+09 _runtime‚13448.188924583 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1037941321973248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚922397770750609 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚909659632460112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12785937209856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚953627607248896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13350619683840 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 516187275 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282643935 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 516213621 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12738138290497 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚953628535223296 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 233569686 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚953627607248896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 233569684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282617591 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12737230315493 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139953454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12785027238400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚940842580010496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1037941298033152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1051291684740608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 233569686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚940842598013440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚940842598013440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207677577 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 516232266 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139965517 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚940924199096320 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207694640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1038031523737600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12739255563983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12786148698624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 516258638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚940924217099264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233589844 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1051383084930048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚953711275782656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚922477905618639 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12738347588771 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚922476980604067 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12787058683392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12786148698624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1038031547677696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67729123 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13351783669760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13351561192448 _runtime‚13449.350738329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚940924199096320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282642422 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207695261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233589846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282642422 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚953710347794944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909738650054656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282668792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12787058683392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 516232266 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 516232266 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139965517 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚940924199096320 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207694640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1038031523737600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12739255563983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12786148698624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 516258638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚940924217099264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233589844 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1051383084930048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚953711275782656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚922477905618639 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12738347588771 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚922476980604067 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12787058683392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12786148698624 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1038031547677696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67729123 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13351783669760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13351561192448 _runtime‚13449.350738329 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚940924199096320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282642422 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207695261 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233589846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282642422 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚953710347794944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909738650054656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282668792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12787058683392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 516232266 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚11199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚13508.964210107 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 234577684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68011248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚957765576939520 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042452583255552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚913610512187312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234577686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208567952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚957765576939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68010641 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926404527732917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚944923534301696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚944923534301696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283888837 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283862493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12793107570601 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1055861810621952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1042452607195648 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 234577686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 283862493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚944923552304640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 234577684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12842024634880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚957764648965120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12842024634880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12841114663424 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 283888837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12841114663424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0.®G=(Ç-‹P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚944923534301696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚944923534301696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283888837 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283862493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12793107570601 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1055861810621952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1042452607195648 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 234577686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 283862493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚944923552304640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 234577684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12842024634880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚957764648965120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12842024634880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12841114663424 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 283888837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12841114663424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885381254249868e+09 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1055861559996928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚913610495147952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 518440177 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208567343 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12794015545605 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚11199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚13508.964210107 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 234577684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68011248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚957765576939520 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042452583255552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚913610512187312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234577686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208567952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚957765576939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68010641 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926404527732917˜Ûœ4È-/ _wandbruntime‚13510  _runtime‚13510¦e@ 'É-N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12843146108416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚957848317498880 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208585120 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1055952945495040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518511540 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12794224843879 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283887324 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926483737586375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518485168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚945005153387520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13410369498112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚957848317498880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 234597846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 140568767 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957847389511168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 518485168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234597844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13510.180193457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518511540 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚926484662600947 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12842236123648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚921 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚945005153387520 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140568769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283887324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208585764ŽB¸þÊ-•N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926483737586375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518485168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚945005153387520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on9Cn§%_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13410369498112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚957848317498880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 234597846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 140568767 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957847389511168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 518485168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234597844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13510.180193457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518511540 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚926484662600947 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12842236123648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚921 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚945005153387520 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140568769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283887324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208585764 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12843146108416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚957848317498880 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208585120 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1055952945495040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518511540 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12794224843879 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283887324 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0™úÉ®4Ë-/ _wandbruntime‚13510  _runtime‚13510 Óºï_Ì-Ù 1loss/FBt2/assignment_loss‚0.16269721067677193 0loss/SRqqt1/detection_loss‚2.388570796359669 _runtime‚13510.181698984 2loss/FBt3/assignment_loss‚0.028553678532173336 .loss/FRt1/detection_loss‚2.669081482195085 /loss/FBt3/detection_loss‚0.6022211132083266 /loss/FRt1/assignment_loss‚2.468521777660616 0loss/FRt3/assignment_loss‚1.2151385612890755 1loss/SRqqt2/assignment_loss‚1.488957038883007 1loss/FBt4/assignment_loss‚0.45480817976661353 1loss/SRqqt2/detection_loss‚2.1729275809996054 %loss/total_loss‚37.57392840213726 /loss/FRt4/assignment_loss‚2.649215622672013 /loss/FRt2/detection_loss‚2.4886079272556834 1loss/SRqqt4/detection_loss‚2.1920853843912482 epoch‚9 trainer/global_step‚11199 % _timestamp‚1.7885381266463678e+09 _step‚922 1loss/FBt1/assignment_loss‚0.15249540338623369 .loss/FBt1/detection_loss‚2.452719602319929 /loss/FBt4/detection_loss‚2.0798285534748664 1loss/SRqqt3/detection_loss‚0.8064665588151131 2loss/SRqqt3/assignment_loss‚0.8414538129356367 .loss/FRt4/detection_loss‚2.117261098165597 /loss/FBt2/detection_loss‚1.7304198086261748 0loss/FRt2/assignment_loss‚3.0478384494781494 2loss/SRqqt1/assignment_loss‚0.9579927495426752 2loss/SRqqt4/assignment_loss‚1.1239199305418879 /loss/FRt3/detection_loss‚1.2821460809611098é¼dÍ-Þ 0loss/FRt2/assignment_loss‚3.0478384494781494 2loss/SRqqt1/assignment_loss‚0.9579927495426752 2loss/SRqqt4/assignment_loss‚1.1239199305418879 /loss/FRt3/detection_loss‚1.2821460809611098 1loss/FBt2/assignment_loss‚0.16269721067677193 0loss/SRqqt1/detection_loss‚2.388570796359669 _runtime‚13510.181698984 2loss/FBt3/assignment_loss‚0.028553678532173336 .loss/FRt1/detection_loss‚2.669081482195085 /loss/FBt3/detection_loss‚0.6022211132083266 /loss/FRt1/assignment_loss‚2.468521777660616 0loss/FRt3/assignment_loss‚1.2151385612890755 1loss/SRqqt2/assignment_loss‚1.488957038883007 1loss/FBt4/assignment_loss‚0.45480817976661353 1loss/SRqqt2/detection_loss‚2.1729275809996054 %loss/total_loss‚37.57392840213726 /loss/FRt4/assignment_loss‚2.649215622672013 /loss/FRt2/detection_loss‚2.4886079272556834 1loss/SRqqt4/detection_loss‚2.1920853843912482 epoch‚9 trainer/global_step‚11199 % _timestamp‚1.7885381266463678e+09 _step‚922 1loss/FBt1/assignment_loss‚0.15249540338623369 .loss/FBt1/detection_loss‚2.452719602319929 /loss/FBt4/detection_loss‚2.0798285534748664 1loss/SRqqt3/detection_loss‚0.8064665588151131 2loss/SRqqt3/assignment_loss‚0.8414538129356367 .loss/FRt4/detection_loss‚2.117261098165597 /loss/FBt2/detection_loss‚1.7304198086261748š¥Æ\’Î-‚(:‡ “ÚëÔÛ‡¸à gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó·Áë|Ï-‚(:ñ “ÚëÔá­¨á disk.sda1.out‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1961.55126953125 disk.md0p1.out‚ 1907.4375! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.09144314601127% disk./.usageGB‚2721.3985748291016 disk.sdb1.in‚0% proc.memory.availableMB‚ 1307.1875 disk.rbd0.out‚ 4.16259765625 disk.sda1.in‚0 disk.dm-0.out‚218720.4140625 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.in‚ 208.765625 disk.rbd2.in‚0$ disk.nvme0n1.out‚153962.16015625 disk.dm-0.in‚ 8532.8125! disk.sdb.out‚1961.55126953125 disk.sdb2.in‚ 75.59765625$ memory_percent‚92.02156066894531 cpu‚25.231567842904962 disk.md0.out‚ 1907.4375! disk.sda.out‚1961.55126953125 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 3.014067e+07 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625" disk.sdb2.out‚1961.55126953125# disk.nvme1n1.out‚64758.25390625 disk.rbd0.in‚0 network.recv‚ 4.495387e+06 disk.rbd2.out‚ 4.150390625! disk.nvme0n1.in‚ 7682.55078125 disk.sda.in‚ 133.16796875Ê8M.‘Ð-‚(:† ¢ÚëÔ”ù­à gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùªò¯yÑ-‚(:î ¢ÚëÔ¾ÂÆà disk.sda1.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2977.140625 cpu‚25.231351455311316 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚154049.39453125 disk.md0.out‚ 1908.84375 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1963.017578125 disk.rbd0.in‚0 disk.rbd0.out‚ 4.22509765625 network.recv‚ 4.502201e+06 disk.sdb2.in‚ 75.59765625" disk.nvme1n1.out‚ 64887.8515625 disk.md0.in‚ 208.765625) disk./.usagePercent‚76.06207352852836 disk.rbd2.out‚ 4.150390625* proc.memory.percent‚18.171024322509766 disk.sdb2.out‚1963.017578125 disk.sda2.in‚ 133.16796875 disk.sda2.out‚1963.017578125 disk.dm-0.in‚ 8532.8125 network.sent‚ 3.0235503e+07 disk.sda.in‚ 133.16796875! disk.nvme0n1.in‚ 7682.55078125# memory_percent‚92.0215368270874 proc.cpu.threads‚17$ disk./.usageGB‚2720.348175048828 disk.sda.out‚1963.017578125 disk.md0p1.out‚ 1908.84375! disk.dm-0.out‚218937.24609375) proc.memory.availableMB‚ 1307.19140625 disk.nvme1n1.in‚ 850.26171875 disk.md0p1.in‚ 208.765625~Ò2’Ò-‚(:‡ ±ÚëÔ’êØà gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨!& |Ó-‚(:ñ ±ÚëÔƒŸ”á disk.sda1.out‚0 disk.dm-0.in‚ 8532.8125# disk.nvme1n1.out‚64925.48828125! proc.memory.rssMB‚ 2977.140625 disk.nvme1n1.in‚ 850.26171875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1910.171875% proc.memory.availableMB‚ 1307.1875 disk.rbd0.in‚0 disk.sdb2.in‚ 75.59765625) disk./.usagePercent‚76.05445487337944 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1910.171875 disk.sda1.in‚0$ disk.nvme0n1.out‚154049.58203125 disk.sdb.in‚ 75.59765625 cpu‚25.231111758418002 network.sent‚ 3.0244047e+07 disk.rbd0.out‚ 4.22509765625 disk.rbd2.in‚0 disk.sdb.out‚1964.4111328125 disk.sda.in‚ 133.16796875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 network.recv‚ 4.506837e+06! disk.nvme0n1.in‚ 7682.55078125! disk.sda2.out‚1964.4111328125 disk.md0.in‚ 208.765625 disk.dm-0.out‚218975.0703125! disk.sdb2.out‚1964.4111328125$ disk./.usageGB‚2720.075695037842 disk.rbd2.out‚ 4.150390625 disk.sda2.in‚ 133.16796875 disk.sda.out‚1964.4111328125$ memory_percent‚92.021560668945317d.A’Ô-‚(:‡ ÀÚëÔµ à gpu.0.gpu‚42.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãi%WyÕ-‚(:î ÀÚëÔ‡ÃÓà disk.dm-0.out‚218977.4921875 disk.sdb.in‚ 75.59765625 disk.sdb.out‚1966.4326171875! disk.sdb2.out‚1966.4326171875 disk.sda.out‚1966.4326171875# disk.nvme1n1.out‚64926.23828125 disk.rbd0.out‚ 4.2490234375 network.recv‚ 4.511234e+06) disk./.usagePercent‚76.05205522472777 disk.dm-0.in‚ 8532.8125 disk.sdb1.in‚0 disk.rbd2.out‚ 4.150390625 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚154051.25390625! disk.sda2.out‚1966.4326171875 disk.md0p1.out‚1912.125$ memory_percent‚92.02311038970947 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26171875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1306.93359375 disk.md0.in‚ 208.765625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7682.55078125% disk./.usageGB‚2719.9898719787598 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 network.sent‚ 3.025021e+07 disk.sda2.in‚ 133.16796875! proc.memory.rssMB‚ 2977.140625 cpu‚25.230876919258826 proc.cpu.threads‚17 disk.md0.out‚1912.125W²m4Ö-/ _wandbruntime‚13562  _runtime‚13562ágÊË×-Å ' lr-AdamW/pg1‚1.0733746941459417e-05 ' lr-AdamW/pg2‚1.0733746941459417e-05 trainer/global_step‚11249 % _timestamp‚1.7885381785206356e+09 _runtime‚13562.056134364 _step‚923ö¦¦ÐØ-Ê trainer/global_step‚11249 % _timestamp‚1.7885381785206356e+09 _runtime‚13562.056134364 _step‚923 ' lr-AdamW/pg1‚1.0733746941459417e-05 ' lr-AdamW/pg2‚1.0733746941459417e-05›°áç4Ù-/ _wandbruntime‚13562  _runtime‚13562Ñ— (Ú-†P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 285107395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209460182 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 141159954 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 235585684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚961901690681344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚961902618655744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285107395 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949004488592896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1046963868477952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917561374875152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12849892800713 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚13562.057153099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚961902618655744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12898112059904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1046963892418048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 235585686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917561391914512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285133739 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚949004506595840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12897202088448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚930411284715225 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12897202088448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885381785213685e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949004488592896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209459575 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949004506595840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12848984825709 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚961901690681344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚961902618655744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285107395 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949004488592896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1046963868477952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917561374875152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12849892800713 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚13562.057153099 N@DeviceStatsMonitor.on_train_batch_stara|Gãt/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚961902618655744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12898112059904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1046963892418048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 235585686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917561391914512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285133739 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚949004506595840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12897202088448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚930411284715225 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12897202088448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885381785213685e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949004488592896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209459575 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949004506595840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12848984825709 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141172017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520738070 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235605844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚961985359215104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047054118122496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235605844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚925 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949086107678720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68305343 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520764442 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047054094182400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060522964585472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930490494568683 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 $ _timestamp‚1.788538179474438e+09 _runtime‚13563.010476199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13468611402752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚917640409509056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209477360 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060522705585152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12850102098987 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12899233533440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949086125681664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12899233533440\ÄÉ['Þ-”N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 285158596 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209477985 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12898323548672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 285158596 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141172019 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285132226 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚949086107678720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285132226 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13468846462976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 235605846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚925 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949086107678720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68305343 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520764442 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047054094182400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060522964585472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930490494568683 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 $ _timestamp‚1.788538179474438e+09 _runtime‚13563.010476199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13468611402752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚917640409509056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209477360 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060522705585152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12850102098987 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12899233533440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949086125681664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12899233533440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141172017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520738070 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235605844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚961985359215104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047054118122496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235605844 P¼¦£«265625" disk.nvme1n1.out‚ 64931.3203125 disk.sdb1.in‚0# memory_percent‚92.0217752456665 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625 disk.sda1.out‚0i0©s“æ-‚(:ˆ íÚëÔųÅà gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xdKålç-‚(:á íÚëÔÕȪá! proc.memory.rssMB‚ 2977.140625% disk./.usageGB‚2720.2994804382324 disk.md0p1.out‚ 1916.4765625 disk.sda2.in‚ 133.16796875* proc.memory.percent‚18.171024322509766 cpu‚25.230112024651316 proc.cpu.threads‚17 disk.rbd2.out‚ 4.150390625 disk.nvme0n1.out‚ 154353.5625 disk.md0.out‚ 1916.4765625 disk.dm-0.in‚ 8532.81640625 disk.sdb.in‚ 75.59765625 network.sent‚ 3.0369469e+07 disk.rbd0.in‚0 disk.sdb1.in‚0 network.recv‚ 4.526718e+06) disk./.usagePercent‚76.06071200683583 disk.sda.in‚ 133.16796875 disk.sdb2.out‚ 1970.9609375$ memory_percent‚92.02196598052979 disk.nvme1n1.out‚ 64931.6875 disk.md0p1.in‚ 208.765625 disk.dm-0.out‚ 219285.25 disk.sdb.out‚ 1970.9609375 disk.sdb1.out‚0 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sda2.out‚ 1970.9609375 disk.sda.out‚ 1970.9609375 disk.nvme1n1.in‚ 850.265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚ 4.31201171875! disk.nvme0n1.in‚ 7682.55078125) proc.memory.availableMB‚ 1307.12109375gLT°4è-/ _wandbruntime‚13617  _runtime‚13617ü)eÈé- trainer/global_step‚11299 $ _timestamp‚1.788538234437824e+09 _runtime‚13617.973345026 _step‚927 & lr-AdamW/pg1‚9.500768527853067e-06 & lr-AdamW/pg2‚9.500768527853067e-06«¼XÇÍê-Ç _runtime‚13617.973345026 _step‚927 & lr-AdamW/pg1‚9.500768527853067e-06 & lr-AdamW/pg2‚9.500768527853067e-06 trainer/global_step‚11299 $ _timestamp‚1.788538234437824e+09Ÿ“^f54ë-/ _wandbruntime‚13617  _runtime‚13617ïÚðh (ì-†P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68589317 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885382344389126e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 236593686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12954199484928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13525998212096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 522945981 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12905770055821 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12904862080817 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚966039660371968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13526218605568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚966038732397568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚921512271641712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12953289513472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚953085460887040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1051475153700352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚921512254602352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1065001151912448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934418041697533 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚13617.974510351 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 141763204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚953085460887040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236593686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210352521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210351903 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286378641 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚953085442884096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12954199484928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 236593684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12953289513472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966038732397568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051475177640448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934417116683169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286352297 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286352297 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141763202 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1065001396246016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚953085460887040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236593686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210352521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210351903 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286378641 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚953085442884096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12954199484928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 236593684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12953289513472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966038732397568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051475177640448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934417116683169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286352297 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286352297 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141763202 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1065001396246016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236613844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286403498 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12954410973696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚966122400931328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12905979354095 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210369765 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522990972 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13527405426688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523017344 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1065092808771584 $ _timestamp‚1.788538235521119e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13527187143680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934497251550991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141775269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286377128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210370370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065092566548480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12954410973696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚953167079972864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12955320958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286377128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236613846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051565379404800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚966121472943616 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 Z97”YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚953167061969920aÛ=+'ð-“N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141775267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚966122400931328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236613844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286403498 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12954410973696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚966122400931328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12905979354095 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210369765 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522990972 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13527405426688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523017344 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1065092808771584 $ _timestamp‚1.788538235521119e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13527187143680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934497251550991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141775269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286377128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210370370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065092566548480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12954410973696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚953167079972864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12955320958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286377128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236613846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051565379404800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚966121472943616 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚953167061969920¡e«¼4ñ-/ _wandbruntime‚13619  _runtime‚13619"1=Zò-Ô /loss/FRt3/assignment_loss‚1.283451680953686 0loss/SRqqt2/detection_loss‚2.042658540109793 .loss/FBt2/detection_loss‚2.430373953921454 .loss/FRt1/detection_loss‚3.743645902474721 /loss/FRt2/detection_loss‚2.5203497969598496 1loss/SRqqt4/detection_loss‚2.3661123832066853 $ _timestamp‚1.788538235524295e+09 /loss/FRt1/assignment_loss‚3.481444303194682 2loss/SRqqt3/assignment_loss‚0.7425433426150155 1loss/SRqqt4/assignment_loss‚1.276944954196612 2loss/FBt3/assignment_loss‚0.037533349692821315 .loss/FBt1/detection_loss‚2.043986614545186 .loss/FBt3/detection_loss‚0.560704302340746 trainer/global_step‚11299 /loss/FRt4/assignment_loss‚3.178142489456549 2loss/SRqqt2/assignment_loss‚1.4709589404260948 1loss/FBt1/assignment_loss‚0.13555464382446492 %loss/total_loss‚41.38067890588919 _step‚930 1loss/FBt4/assignment_loss‚0.13511922906571333 _runtime‚13619.059631515 epoch‚9 0loss/FRt2/assignment_loss‚3.3018471283071182 1loss/FBt2/assignment_loss‚0.19224804639816284 /loss/FRt3/detection_loss‚1.3400521410199313 /loss/FRt4/detection_loss‚2.2904114226651626 1loss/SRqqt3/detection_loss‚0.6575474475617878 .loss/FBt4/detection_loss‚2.235474369742654 2loss/SRqqt1/assignment_loss‚1.3626490744451683 0loss/SRqqt1/detection_loss‚2.550924848765135¶ Â_ó-Ù .loss/FRt1/detection_loss‚3.743645902474721 /loss/FRt2/detection_loss‚2.5203497969598496 1loss/SRqqt4/detection_loss‚2.3661123832066853 $ _timestamp‚1.788538235524295e+09 /loss/FRt1/assignment_loss‚3.481444303194682 2loss/SRqqt3/assignment_loss‚0.7425433426150155 1loss/SRqqt4/assignment_loss‚1.276944954196612 2loss/FBt3/assignment_loss‚0.037533349692821315 .loss/FBt1/detection_loss‚2.043986614545186 .loss/FBt3/detection_loss‚0.560704302340746 trainer/global_step‚11299 /loss/FRt4/assignment_loss‚3.178142489456549 2loss/SRqqt2/assignment_loss‚1.4709589404260948 1loss/FBt1/assignment_loss‚0.13555464382446492 %loss/total_loss‚41.38067890588919 _step‚930 1loss/FBt4/assignment_loss‚0.13511922906571333 _runtime‚13619.059631515 epoch‚9 0loss/FRt2/assignment_loss‚3.3018471283071182 1loss/FBt2/assignment_loss‚0.19224804639816284 /loss/FRt3/detection_loss‚1.3400521410199313 /loss/FRt4/detection_loss‚2.2904114226651626 1loss/SRqqt3/detection_loss‚0.6575474475617878 .loss/FBt4/detection_loss‚2.235474369742654 2loss/SRqqt1/assignment_loss‚1.3626490744451683 0loss/SRqqt1/detection_loss‚2.550924848765135 /loss/FRt3/assignment_loss‚1.283451680953686 0loss/SRqqt2/detection_loss‚2.042658540109793 .loss/FBt2/detection_loss‚2.430373953921454¢Bùþµ”ô-‚(:‰ üÚëÔ·ñúÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 1i(ˆõ-‚(:ý üÚëÔé™âá disk.sdb.in‚ 75.59765625" disk.sdb2.out‚1972.87451171875 disk.nvme1n1.in‚ 850.265625 network.sent‚ 3.0373412e+07 disk.dm-0.in‚ 8532.81640625 disk.sda.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1918.32421875 disk.sdb1.out‚0$ disk.nvme0n1.out‚154470.48046875" disk.sda2.out‚1972.87451171875* proc.memory.percent‚18.171024322509766 cpu‚25.229820742529544 disk.dm-0.out‚219402.3203125! disk.sda.out‚1972.87451171875 disk.sda1.out‚0 disk.rbd2.out‚ 4.150390625 disk.md0.out‚ 1918.32421875& proc.memory.availableMB‚ 1306.59375 disk.rbd0.in‚0 disk.rbd0.out‚ 4.31201171875 disk.sdb1.in‚0) disk./.usagePercent‚76.06770947104567 disk.sda1.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2977.140625 network.recv‚ 4.530825e+06! disk.sdb.out‚1972.87451171875# disk.nvme1n1.out‚64931.83984375 disk.sda2.in‚ 133.16796875$ memory_percent‚92.02518463134766 proc.cpu.threads‚17! disk.nvme0n1.in‚ 7682.55078125% disk./.usageGB‚2720.5497436523438ÀêÚ@”ö-‚(:‰ ‹ÛëÔèÙÞ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä¿Mé}÷-‚(:ò ‹ÛëÔõñÐß* proc.memory.percent‚18.171024322509766 cpu‚25.229613146633916) disk./.usagePercent‚76.07119545721702 disk.sda.out‚1974.3505859375 disk.rbd2.in‚0 disk.sdb.out‚1974.3505859375 disk.md0p1.in‚ 208.765625! disk.nvme0n1.in‚ 7682.55078125! disk.sda2.out‚1974.3505859375 disk.nvme1n1.in‚ 850.265625$ disk.nvme0n1.out‚154639.52734375 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8532.81640625! proc.memory.rssMB‚ 2977.140625$ disk./.usageGB‚2720.674419403076 disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚1974.3505859375 network.recv‚ 4.537653e+06 disk.sdb1.in‚0! disk.dm-0.out‚219571.94140625 disk.sdb2.in‚ 75.59765625$ memory_percent‚92.02733039855957 proc.cpu.threads‚17 network.sent‚ 3.0480652e+07" disk.nvme1n1.out‚ 64932.4140625 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.md0.out‚ 1919.73046875( proc.memory.availableMB‚ 1306.2421875 disk.rbd0.out‚4.375 disk.rbd2.out‚ 4.150390625 disk.sdb.in‚ 75.59765625 disk.md0p1.out‚ 1919.73046875 disk.sda1.out‚0¯î“ø-‚(:ˆ šÛëÔýЇß gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚w€ù-‚(:õ šÛëÔ쮹à disk.sdb2.in‚ 75.59765625 disk.sda1.out‚0! disk.nvme0n1.in‚ 7682.55078125) disk./.usagePercent‚76.08517715973072 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.md0.out‚ 1921.33984375 disk.dm-0.out‚219740.8046875 disk.sdb.in‚ 75.59765625( proc.memory.availableMB‚ 1306.2109375 network.sent‚ 3.0486446e+07 network.recv‚ 4.541622e+06 disk.nvme1n1.in‚ 850.265625 disk.sdb1.out‚0 disk.rbd2.out‚ 4.150390625$ memory_percent‚92.02752113342285 cpu‚25.22939472407991 disk.rbd0.out‚4.375$ disk.nvme0n1.out‚154808.26171875! disk.sdb.out‚1976.02783203125! disk.sda.out‚1976.02783203125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0" disk.sdb2.out‚1976.02783203125 disk.md0p1.out‚ 1921.33984375" disk.sda2.out‚1976.02783203125 disk.md0.in‚ 208.765625* proc.memory.percent‚18.171024322509766$ disk./.usageGB‚2721.174472808838 disk.sda2.in‚ 133.16796875! proc.memory.rssMB‚ 2977.140625# disk.nvme1n1.out‚64932.54296875 disk.dm-0.in‚ 8532.81640625 proc.cpu.threads‚17P=•Š“ú-‚(:ˆ ©ÛëÔ„‡Àà gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ‹#¦{û-‚(:ð ©ÛëÔñÄŠâ$ disk./.usageGB‚2721.174415588379 disk.rbd2.in‚0 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625! disk.sdb2.out‚1977.7919921875 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 network.sent‚ 3.0494872e+07 network.recv‚ 4.546567e+06 disk.sdb.out‚1977.7919921875 disk.md0p1.out‚ 1923.0390625 disk.rbd0.in‚0 disk.rbd0.out‚4.375! disk.nvme0n1.in‚ 7682.55078125 disk.sdb1.out‚0 disk.md0.out‚ 1923.0390625 disk.dm-0.in‚ 8532.81640625 disk.dm-0.out‚219910.0078125$ memory_percent‚92.02611446380615) proc.memory.availableMB‚ 1306.44140625! proc.memory.rssMB‚ 2977.140625 disk.sda1.in‚0 disk.rbd2.out‚ 4.150390625 disk.sdb1.in‚0! disk.sda2.out‚1977.7919921875 disk.sda.out‚1977.7919921875 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 cpu‚25.22919185509226) disk./.usagePercent‚76.08517555982274 disk.sda1.out‚0$ disk.nvme0n1.out‚154977.16015625# disk.nvme1n1.out‚64932.84765625* proc.memory.percent‚18.171024322509766 disk.nvme1n1.in‚ 850.265625\ÁÏ÷4ü-/ _wandbruntime‚13676  _runtime‚13676ß dµÉý-à _step‚931 & lr-AdamW/pg1‚8.342063125189965e-06 & lr-AdamW/pg2‚8.342063125189965e-06 trainer/global_step‚11349 % _timestamp‚1.7885382931046247e+09 _runtime‚13676.640272742Ÿ‘Îþ-È & lr-AdamW/pg1‚8.342063125189965e-06 & lr-AdamW/pg2‚8.342063125189965e-06 trainer/global_step‚11349 % _timestamp‚1.7885382931046247e+09 _runtime‚13676.640272742 _step‚931£¨Óýr4ÿ-/ _wandbruntime‚13676  _runtime‚13676_í§ (€.†P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1069571204096000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 142366454 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12960739335925 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055986438922752 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287597199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970176702088192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287623543 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _runtime‚13676.641672838 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970175774113792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211245684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055986462862848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13009376938496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970175774113792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚957166415178240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _step‚932 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 237601686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 237601684 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1069571461012480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚957166415178240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13584765173248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142366452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13009376938496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13010286909952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287623543 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885382931058095e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚957166397175296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970176702088192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚957166397175296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211245056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 525225229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142366452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13009376938496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13010286909952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287623543 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885382931058095e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚957166397175296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970176702088192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚957166397175296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211245056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 525225229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055986438922752 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287597199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970176702088192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287623543 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _runtime‚13676.641672838 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970175774113792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211245684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055986462862848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13009376938496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970175774113792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¤“û«4‚./ _wandbruntime‚13677  _runtime‚13677zc_'ƒ.N PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 525270246 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142378519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 287648400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970259442647552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚957248016261120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13010498398720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚970258514659840 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237621846 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211262944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925542168963456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚925542151924096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885382941837008e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 525243874 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 237621844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12961856609203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68884427 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 525270246 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13010498398720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 287622030 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211263568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13011408383488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68885049 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056076664627200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚13677.719899811 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13586159193600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚957248034264064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 525243874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1069662847760896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287648400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12962764584415 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056076688567296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1069662595052032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 trainer/global_step‚11349ß±Ûµ'„.•N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1069662595052032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 trainer/global_step‚11349 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 525270246 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142378519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 287648400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970259442647552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚957248016261120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13010498398720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚970258514659840 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237621846 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211262944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925542168963456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚925542151924096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885382941837008e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 525243874 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 237621844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12961856609203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68884427 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 525270246 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13010498398720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 287622030 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211263568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13011408383488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68885049 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056076664627200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚13677.719899811 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13586159193600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚957248034264064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 525243874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1069662847760896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287648400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12962764584415 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056076688567296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615¥©+€ 4…./ _wandbruntime‚13677  _runtime‚13677ì–Ť_†.Ù 0loss/FRt2/assignment_loss‚3.4544368231738054 1loss/SRqqt2/assignment_loss‚1.334326524337804 /loss/FRt3/detection_loss‚1.6263981434702872 %loss/total_loss‚40.62107932720414 _step‚934 1loss/FBt2/assignment_loss‚0.33161153807304733 1loss/FBt4/assignment_loss‚0.09720505175939041 /loss/FBt1/detection_loss‚2.1165492713451384 trainer/global_step‚11349 2loss/SRqqt3/assignment_loss‚0.8497558152809254 2loss/SRqqt4/assignment_loss‚1.4515284463190115 0loss/SRqqt4/detection_loss‚2.177948604791592 0loss/FBt3/detection_loss‚0.48320110282477213 1loss/SRqqt1/assignment_loss‚1.267334553054892 2loss/FBt3/assignment_loss‚0.025780271639985714 /loss/FRt4/detection_loss‚2.3113137721651937 1loss/SRqqt3/detection_loss‚0.7590303630568087 0loss/FRt1/assignment_loss‚2.7593354412487576 /loss/FRt3/assignment_loss‚1.139549663066864 /loss/FRt2/detection_loss‚2.6320629506820263 0loss/SRqqt2/detection_loss‚2.433459898910007 /loss/FRt4/assignment_loss‚2.472790973942454 /loss/FRt1/detection_loss‚3.5076256905283247 1loss/SRqqt1/detection_loss‚2.5326440815742197 /loss/FBt2/detection_loss‚2.3281756829131734 .loss/FBt4/detection_loss‚2.285905273337113 _runtime‚13677.721436098 0loss/FBt1/assignment_loss‚0.2431093897085475 epoch‚9 % _timestamp‚1.7885382941861634e+09y=1d‡.Þ 1loss/SRqqt1/assignment_loss‚1.267334553054892 2loss/FBt3/assignment_loss‚0.025780271639985714 /loss/FRt4/detection_loss‚2.3113137721651937 1loss/SRqqt3/detection_loss‚0.7590303630568087 0loss/FRt1/assignment_loss‚2.7593354412487576 /loss/FRt3/assignment_loss‚1.139549663066864 /loss/FRt2/detection_loss‚2.6320629506820263 0loss/SRqqt2/detection_loss‚2.433459898910007 /loss/FRt4/assignment_loss‚2.472790973942454 /loss/FRt1/detection_loss‚3.5076256905283247 1loss/SRqqt1/detection_loss‚2.5326440815742197 /loss/FBt2/detection_loss‚2.3281756829131734 .loss/FBt4/detection_loss‚2.285905273337113 _runtime‚13677.721436098 0loss/FBt1/assignment_loss‚0.2431093897085475 epoch‚9 % _timestamp‚1.7885382941861634e+09 0loss/FRt2/assignment_loss‚3.4544368231738054 1loss/SRqqt2/assignment_loss‚1.334326524337804 /loss/FRt3/detection_loss‚1.6263981434702872 %loss/total_loss‚40.62107932720414 _step‚934 1loss/FBt2/assignment_loss‚0.33161153807304733 1loss/FBt4/assignment_loss‚0.09720505175939041 /loss/FBt1/detection_loss‚2.1165492713451384 trainer/global_step‚11349 2loss/SRqqt3/assignment_loss‚0.8497558152809254 2loss/SRqqt4/assignment_loss‚1.4515284463190115 0loss/SRqqt4/detection_loss‚2.177948604791592 0loss/FBt3/detection_loss‚0.48320110282477213¦tkÂL”ˆ.‚(:‰ ¸ÛëÔù»ˆá gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–|«ë}‰.‚(:ò ¸ÛëÔÝ­¤á network.sent‚ 3.0498644e+07 disk.dm-0.out‚220078.8671875" disk.sdb2.out‚1979.26416015625 disk.sdb1.in‚0 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.md0.out‚ 1924.4453125! disk.sda.out‚1979.26416015625 disk.dm-0.in‚ 8532.81640625 disk.sda1.in‚0$ memory_percent‚92.02663898468018* proc.memory.percent‚18.171024322509766 disk.md0p1.out‚ 1924.4453125 disk.sda1.out‚0! disk.nvme0n1.in‚ 7682.55078125! proc.memory.rssMB‚ 2977.140625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.nvme1n1.in‚ 850.265625) disk./.usagePercent‚76.09920963265772! disk.sdb.out‚1979.26416015625 disk.sda.in‚ 133.16796875 disk.md0.in‚ 208.765625 cpu‚25.228998867368176 disk.rbd0.out‚4.375 disk.rbd2.out‚ 4.150390625 proc.cpu.threads‚17" disk.nvme1n1.out‚ 64932.9765625 network.recv‚ 4.550296e+06 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1979.26416015625) proc.memory.availableMB‚ 1306.35546875$ disk./.usageGB‚2721.676342010498 disk.sdb.in‚ 75.59765625 disk.rbd0.in‚0" disk.nvme0n1.out‚ 155145.890625µ]䙞Š.‚(:“ ÇÛëÔŽ¡âà gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÑh`Z‹.‚(:ó ÇÛëÔ¶Ÿ·â" disk.sda2.out‚1980.84521484375 disk.sdb2.in‚ 75.59765625 disk.sdb1.out‚0$ memory_percent‚92.02523231506348 cpu‚25.228780449959565 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0) disk./.usagePercent‚76.09915160932825 disk.dm-0.out‚220365.2265625 disk.sdb.in‚ 75.59765625! disk.sdb`÷J$.out‚1980.84521484375# disk.nvme1n1.out‚64933.55078125% disk./.usageGB‚2721.6742668151855 disk.sda2.in‚ 133.16796875" disk.sdb2.out‚1980.84521484375 disk.nvme1n1.in‚ 850.265625 disk.rbd0.out‚4.375! proc.memory.rssMB‚ 2977.140625! disk.nvme0n1.in‚ 7682.55078125 disk.md0.in‚ 208.765625 disk.md0p1.out‚ 1925.953125 disk.rbd2.in‚0( proc.memory.availableMB‚ 1306.5859375* proc.memory.percent‚18.171024322509766$ disk.nvme0n1.out‚155431.67578125 disk.rbd0.in‚0 disk.sda1.out‚0 network.sent‚ 3.0607348e+07 disk.md0.out‚ 1925.953125! disk.sda.out‚1980.84521484375 network.recv‚ 4.556923e+06 disk.dm-0.in‚ 8532.81640625 disk.sdb1.in‚0 disk.rbd2.out‚ 4.150390625 proc.cpu.threads‚17 disk.sda.in‚ 133.16796875…ùþ”Œ.‚(:‰ ÖÛëÔ¸˜á gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ x.‚(:í ÖÛëÔËšÛâ disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.sda1.in‚0# disk.nvme1n1.out‚64933.71484375 disk.md0p1.out‚ 1927.4609375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 1927.4609375 disk.dm-0.in‚ 8532.81640625 disk.sdb.out‚1982.4072265625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.265625 cpu‚25.22856601578899 network.recv‚ 4.560625e+06) disk./.usagePercent‚76.09911385149987 disk.rbd2.in‚0 disk.dm-0.out‚220637.7734375 disk.sdb.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625! disk.sda2.out‚1982.4072265625 disk.md0.in‚ 208.765625 disk.sda.out‚1982.4072265625 disk.md0p1.in‚ 208.765625$ memory_percent‚92.02995300292969 network.sent‚ 3.0610786e+07% disk./.usageGB‚2721.6729164123535 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766! disk.nvme0n1.in‚ 7682.55078125 disk.sdb2.in‚ 75.59765625$ disk.nvme0n1.out‚155704.05859375 disk.rbd0.out‚4.375% proc.memory.availableMB‚ 1305.8125 disk.rbd2.out‚ 4.150390625! disk.sdb2.out‚1982.4072265625î(Ž…’Ž.‚(:‡ åÛëÔ•ßÙÞ gpu.0.gpu‚19.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÄ`u|.‚(:ñ åÛëÔùÃÿà disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1929.1328125 disk.rbd0.out‚4.375! disk.nvme0n1.in‚ 7682.55078125! disk.sda2.out‚1984.1298828125 disk.sda.in‚ 133.16796875 cpu‚25.228359526585635 network.recv‚ 4.566095e+06# disk.nvme1n1.out‚65050.31640625 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625 disk.sdb2.in‚ 75.59765625) proc.memory.availableMB‚ 1306.05078125 disk.sdb1.in‚0 disk.sda.out‚1984.1298828125 network.sent‚ 3.0617526e+07 disk.rbd0.in‚0 disk.sdb.in‚ 75.59765625! disk.sdb2.out‚1984.1298828125! proc.memory.rssMB‚ 2977.140625$ disk.nvme0n1.out‚155872.85546875% disk./.usageGB‚2721.7979431152344 disk.dm-0.out‚ 220923.171875 disk.sdb.out‚1984.1298828125 proc.cpu.threads‚17) disk./.usagePercent‚76.10260965044016 disk.nvme1n1.in‚ 850.265625 disk.md0.in‚ 208.765625 disk.md0.out‚ 1929.1328125 disk.dm-0.in‚ 8532.81640625* proc.memory.percent‚18.171024322509766 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875$ memory_percent‚92.02849864959717B¾f4./ _wandbruntime‚13732  _runtime‚13732 Mü;É‘.à & lr-AdamW/pg2‚7.257872205939235e-06 trainer/global_step‚11399 % _timestamp‚1.7885383492863362e+09 _runtime‚13732.821779996 _step‚935 & lr-AdamW/pg1‚7.257872205939235e-06Ò©z Î’.È _runtime‚13732.821779996 _step‚935 & lr-AdamW/pg1‚7.257872205939235e-06 & lr-AdamW/pg2‚7.257872205939235e-06 trainer/global_step‚11399 % _timestamp‚1.7885383492863362e+09§¶¬T—4“./ _wandbruntime‚13732  _runtime‚13732}¸þ• (”.…P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13065464363520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13065464363520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚974312815830016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 238609686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288842101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚974312815830016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942430630647785 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288868445 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142969702 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1074141292360704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074141041735680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527478131 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 238609684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288868445 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13066374334976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚961247351466496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288842101 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 142969704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 238609684 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚929414031096112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚961247369469440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69168233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144á=ãx(•.ŠP D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚961247369469440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69168233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13065464363520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13065464363520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚974312815830016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 238609686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288842101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚974312815830016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942430630647785 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288868445 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142969702 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1074141292360704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074141041735680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527478131 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 238609684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288868445 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13066374334976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚961247351466496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288842101 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 142969704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 238609684 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚929414031096112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0¨ã.®+4–./ _wandbruntime‚13733  _runtime‚13733¶sc›—.N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 288866932 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69174031 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942511690530179 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212155798 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142981767 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 238629844 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13644712051200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13067495808512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚961328988555264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69174652 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1074232685840896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942510765515607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288893302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527496776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 238629846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1060587973789696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13644476990976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527523148 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚974395556376064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1060587949849600 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 527496776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 288866932 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142981769 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1074232426840576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.curU^ìýrent‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13017733864311 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13067495808512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942511690530179 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212155798 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142981767 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 238629844 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13644712051200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13067495808512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚961328988555264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69174652 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1074232685840896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942510765515607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288893302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527496776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 238629846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1060587973789696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13644476990976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527523148 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚974395556376064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1060587949849600 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 527496776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 288866932 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142981769 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1074232426840576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13017733864311 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13067495808512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F‹ÁP}Ÿ.‚(:ò ƒÜëÔ´Ûåà disk.rbd2.out‚ 4.150390625 cpu‚25.227917442987838 network.recv‚ 4.577071e+06! disk.dm-0.out‚221289.58984375 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625 disk.rbd0.out‚4.375 disk.sda1.out‚0$ memory_percent‚92.02723503112793! proc.memory.rssMB‚ 2977.140625! disk.sda.out‚1987.60009765625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1932.484375# disk.nvme0n1.out‚156238.8046875* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.08434648750655 disk.sdb1.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.sda2.in‚ 133.16796875" disk.sda2.out‚1987.60009765625 disk.md0.out‚ 1932.484375" disk.sdb2.out‚1987.60009765625 disk.sda.in‚ 133.16796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.md0.in‚ 208.765625 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sdb.out‚1987.60009765625 network.sent‚ 3.0733474e+07 disk.dm-0.in‚ 8532.81640625# disk.nvme1n1.out‚65050.78515625( proc.memory.availableMB‚ 1306.2578125 proc.cpu.threads‚17$ disk./.usageGB‚2721.144763946533 disk.sdb.in‚ 75.59765625ïi/'“ .‚(:ˆ ’ÜëÔ‰£ß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=V†di¡.‚(:Þ ’ÜëÔ¶™ûá! disk.nvme0n1.in‚ 7682.55078125 disk.nvme0n1.out‚ 156343.125 disk.md0p1.out‚ 1933.96875 disk.rbd0.in‚0 disk.rbd2.out‚ 4.150390625 disk.sda2.out‚ 1989.14453125 disk.md0.out‚ 1933.96875) proc.memory.availableMB‚ 1306.01171875* proc.memory.percent‚18.171024322509766 network.sent‚ 3.0739028e+07) disk./.usagePercent‚76.08276289858614 disk.md0p1.in‚ 208.765625 disk.sdb2.in‚ 75.59765625$ memory_percent‚92.02873706817627 disk.sda.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625 cpu‚25.227683331059946 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚ 221394.078125 disk.sdb2.out‚ 1989.14453125! proc.memory.rssMB‚ 2977.140625 disk.sda.out‚ 1989.14453125 disk.rbd0.out‚4.375 disk.sdb.out‚ 1989.14453125 disk.sdb1.out‚0 disk.nvme1n1.in‚ 850.265625 disk.sda1.in‚0 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625 network.recv‚ 4.580517e+06% disk./.usageGB‚2721.0881271362305! disk.nvme1n1.out‚ 65050.953125 disk.sdb1.in‚0 disk.sda1.out‚0“H»”¢.‚(:‰ ¡ÜëÔÜÅ‹â gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bE»Œ~£.‚(:ó ¡ÜëÔ¿½ä) disk./.usagePercent‚76.08511327007199! disk.sdb.out‚1991.41748046875! disk.sda.out‚1991.41748046875 disk.rbd0.in‚0* proc.memory.percent‚18.171024322509766 disk.sda2.in‚ 133.16796875 disk.dm-0.out‚221584.1171875 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚4.375 disk.rbd2.in‚0" disk.sda2.out‚1991.41748046875 disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625" disk.nvme1n1.out‚ 65051.1484375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 4.150390625! disk.nvme0n1.in‚ 7682.55078125 disk.md0p1.in‚ 208.765625 disk.sdb1.out‚0$ memory_percent‚92.02873706817627) proc.memory.availableMB‚ 1306.01171875 cpu‚25.227427669052357 network.recv‚ 4.585484e+06 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.md0p1.out‚ 1936.16796875 disk.md0.out‚ 1936.16796875 disk.dm-0.in‚ 8532.81640625" disk.sdb2.out‚1991.41748046875 disk.sdb1.in‚0 network.sent‚ 3.0745404e+07$ disk./.usageGB‚2721.172187805176! disk.nvme0n1.out‚ 156532.96875 disk.sda.in‚ 133.16796875! proc.memory.rssMB‚ 2977.140625Òªˆ=4¤./ _wandbruntime‚13791  _runtime‚13791†±Y¾È¥. $ _timestamp‚1.788538408456462e+09 _runtime‚13791.992122057 _step‚939 & lr-AdamW/pg1‚6.248421713863479e-06 & lr-AdamW/pg2‚6.248421713863479e-06 trainer/global_step‚114492¼%Gͦ.Ç & lr-AdamW/pg1‚6.248421713863479e-06 & lr-AdamW/pg2‚6.248421713863479e-06 trainer/global_step‚11449 $ _timestamp‚1.788538408456462e+09 _runtime‚13791.992122057 _step‚939«&¸º¸4§./ _wandbruntime‚13791  _runtime‚13791j5Sû (¨.†P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978449857546240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚13791.993721384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 239617684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13121551788544 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143572954 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13702217543168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚946437387630093 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529704687 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13122461760000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13072493846141 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290087003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065009009367552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239617684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚965328305757696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚965328323760640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13122461760000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885384084576461e+09 _step‚940 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69458113 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13702001344000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚933364893783952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚965328305757696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239617686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69457476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚946438312644457 trainer/global_step‚11449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978450785520640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213030428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13073401821145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529731033 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143572952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1078711250850816 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239617686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1078711010711552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1065009033307648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933364910823312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290087003 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978449857546240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚13791.993721384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 239617684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13121551788544 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143572954 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13702217543168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚946437387630093 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529704687 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13122461760000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13072493846141 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290087003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065009009367552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239617684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚965328305757696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚965328323760640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13122461760000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885384084576461e+09 _step‚940 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69458113 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13702001344000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚933364893783952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚965328305757696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239617686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69457476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚946438312644457 trainer/global_step‚11449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978450785520640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213030428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13073401821145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529731033 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143572952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1078711250850816 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239617686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1078711010711552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1065009033307648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933364910823312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290087003 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚933443928417856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 239637844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213048943 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚941 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚965409924843520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213048304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529776050 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290138204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69463287 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13074519094631 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚965409942846464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚965409924843520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1078802654833152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13703395821056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143585017 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚946518447512487 _runtime‚13793.225922687 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529749678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚965409942846464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 % _timestamp‚1.7885384096888175e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290111834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1065099235072000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚946517522497915 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239637846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290111834 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572>ë­'¬.•N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚933443911378496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚978532598092288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69463924 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529776050 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290138204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69463287 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13074519094631 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚965409942846464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚965409924843520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1078802654833152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13703395821056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143585017 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚946518447512487 _runtime‚13793.225922687 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529749678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚965409942846464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 % _timestamp‚1.7885384096888175e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290111834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1065099235072000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚946517522497915 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239637846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290111834 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13122673248768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529749678 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13123583233536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚11449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚978533526080000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚933443928417856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 239637844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213048943 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚941 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚965409924843520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213048304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E4­./ _wandbruntime‚13793  _runtime‚13793]êÅ.^®.Ø 2loss/SRqqt4/assignment_loss‚1.7216053392856996 1loss/FBt2/assignment_loss‚0.07411153106555243 1loss/FBt4/assignment_loss‚0.41061106434406575 /loss/FRt2/detection_loss‚2.6133158454652183 .loss/FRt4/detection_loss‚2.024380950523274 1loss/SRqqt2/detection_loss‚2.4825555196514837 1loss/SRqqt3/detection_loss‚0.8451551697756114 1loss/SRqqt4/detection_loss‚1.9317568045981386 _runtime‚13793.228227588 2loss/SRqqt2/assignment_loss‚1.0943982601165771 2loss/SRqqt3/assignment_loss‚0.8862196050878418 0loss/FBt3/detection_loss‚0.42257843427590014 /loss/FRt2/assignment_loss‚2.986651638458515 /loss/FBt4/detection_loss‚1.7096352498782308 .loss/FRt1/assignment_loss‚3.30984911441803 0loss/FRt3/assignment_loss‚0.7975863911888816 1loss/SRqqt1/assignment_loss‚1.375454194523627 %loss/total_loss‚40.20345651455575 _step‚942 % _timestamp‚1.7885384096925693e+09 /loss/FRt4/assignment_loss‚2.874071456948105 /loss/FRt1/detection_loss‚2.9800560903549194 epoch‚9 trainer/global_step‚11449 1loss/FBt3/assignment_loss‚0.11527360265572753 .loss/FBt2/detection_loss‚3.090457077537264 1loss/FBt1/assignment_loss‚0.18826968194727556 /loss/FRt3/detection_loss‚1.4549029841003094 0loss/SRqqt1/detection_loss‚2.401600120589137 /loss/FBt1/detection_loss‚2.4129603877663612uIÌ=c¯.Ý % _timestamp‚1.7885384096925693e+09 /loss/FRt4/assignment_loss‚2.874071456948105 /loss/FRt1/detection_loss‚2.9800560903549194 epoch‚9 trainer/global_step‚11449 1loss/FBt3/assignment_loss‚0.11527360265572753 .loss/FBt2/detection_loss‚3.090457077537264 1loss/FBt1/assignment_loss‚0.18826968194727556 /loss/FRt3/detection_loss‚1.4549029841003094 0loss/SRqqt1/detection_loss‚2.401600120589137 /loss/FBt1/detection_loss‚2.4129603877663612 2loss/SRqqt4/assignment_loss‚1.7216053392856996 1loss/FBt2/assignment_loss‚0.07411153106555243 1loss/FBt4/assignment_loss‚0.41061106434406575 /loss/FRt2/detection_loss‚2.6133158454652183 .loss/FRt4/detection_loss‚2.024380950523274 1loss/SRqqt2/detection_loss‚2.4825555196514837 1loss/SRqqt3/detection_loss‚0.8451551697756114 1loss/SRqqt4/detection_loss‚1.9317568045981386 _runtime‚13793.228227588 2loss/SRqqt2/assignment_loss‚1.0943982601165771 2loss/SRqqt3/assignment_loss‚0.8862196050878418 0loss/FBt3/detection_loss‚0.42257843427590014 /loss/FRt2/assignment_loss‚2.986651638458515 /loss/FBt4/detection_loss‚1.7096352498782308 .loss/FRt1/assignment_loss‚3.30984911441803 0loss/FRt3/assignment_loss‚0.7975863911888816 1loss/SRqqt1/assignment_loss‚1.375454194523627 %loss/total_loss‚40.20345651455575 _step‚942®gç™°.‚(:“ °ÜëÔÖ¥¥à gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 149535457rH^28 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓqÜfm±.‚(:â °ÜëԸݓá disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625# disk.nvme1n1.out‚65051.32421875 disk.nvme0n1.out‚ 156691.4375* proc.memory.percent‚18.171024322509766) disk./.usagePercent‚76.08139231074837$ disk./.usageGB‚2721.039108276367 disk.sda2.out‚1992.708984375$ memory_percent‚92.03052520751953! proc.memory.rssMB‚ 2977.140625 network.sent‚ 3.0762227e+07 network.recv‚ 4.589817e+06 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.sda2.in‚ 133.16796875 disk.dm-0.in‚ 8532.81640625 disk.sdb2.out‚1992.708984375 disk.sda.in‚ 133.16796875 disk.sdb1.out‚0 disk.rbd2.out‚ 4.150390625 disk.md0.in‚ 208.765625 disk.sdb.out‚1992.708984375 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1937.40625 disk.rbd0.out‚4.375 disk.sda.out‚1992.708984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 1937.40625& proc.memory.availableMB‚ 1305.71875 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.dm-0.out‚221742.76171875 cpu‚25.22719222906228 disk.sdb1.in‚0 disk.sdb.in‚ 75.59765625¦&!Ä”².‚(:‰ ¿ÜëÔק–à gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¹¦úp³.‚(:å ¿ÜëÔž£êà) disk./.usagePercent‚76.05931603379142! disk.sdb.out‚1994.18896484375" disk.sdb2.out‚1994.18896484375! disk.sda.out‚1994.18896484375 disk.sdb1.in‚0 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.265625$ memory_percent‚92.02919006347656! disk.nvme0n1.in‚ 7682.55078125 disk.sdb.in‚ 75.59765625 network.sent‚ 3.0858825e+07 disk.rbd0.out‚4.375 disk.sdb1.out‚0# disk.nvme0n1.out‚156811.7734375" disk.sda2.out‚1994.18896484375* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.out‚ 4.150390625% proc.memory.availableMB‚ 1305.9375 disk.rbd2.in‚0 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 network.recv‚ 4.59686e+06# disk./.usageGB‚2720.24955368042 disk.sda1.out‚0 disk.md0.out‚ 1938.8125 disk.dm-0.in‚ 8532.81640625! proc.memory.rssMB‚ 2977.140625 disk.rbd0.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625 disk.dm-0.out‚ 221863.71875" disk.nvme1n1.out‚ 65051.9453125 disk.md0p1.out‚ 1938.8125 cpu‚25.226965287886582 ˜¨’´.‚(:‡ ÎÜëÔÃŽáÞ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`£Wrµ.‚(:ç ÎÜëÔÕ’×â) disk./.usagePercent‚76.06281855234523 disk.md0.out‚ 1940.65234375! disk.nvme1n1.out‚ 65052.109375 disk.md0p1.out‚ 1940.65234375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 850.265625 network.recv‚ 4.60081e+06 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625 disk.sdb2.out‚1996.095703125 disk.rbd0.out‚4.375 disk.dm-0.out‚221928.9921875 disk.md0.in‚ 208.765625 disk.md0p1.in‚ 208.765625 cpu‚25.22671247619728 disk.rbd2.in‚0 disk.dm-0.in‚ 8532.81640625% proc.memory.availableMB‚ 1305.6875 disk.rbd0.in‚0 disk.rbd2.out‚ 4.150390625! disk.nvme0n1.in‚ 7682.55078125 disk.sdb.out‚1996.095703125 disk.sda.in‚ 133.16796875 disk.sda.out‚1996.095703125 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚156876.8828125 disk.sda2.out‚1996.095703125 disk.sdb2.in‚ 75.59765625! proc.memory.rssMB‚ 2977.140625 network.sent‚ 3.0864472e+07% disk./.usageGB‚2720.3748207092285 disk.sdb1.in‚0$ memory_percent‚92.03071594238281* proc.memory.percent‚18.171024322509766dnÕ”“¶.‚(:ˆ ÝÜëÔ‰ÔÅà gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í×w{·.‚(:ð ÝÜëÔçàà# disk.nvme1n1.out‚65053.37109375" disk.sda2.out‚1997.82080078125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚ 75.59765625! disk.dm-0.out‚222028.06640625 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875% proc.memory.availableMB‚ 1306.1875! proc.memory.rssMB‚ 2977.140625 disk.dm-0.in‚ 8532.81640625) disk./.usagePercent‚76.06029901725545 disk.sda2.in‚ 133.16796875" disk.sdb2.out‚1997.82080078125 disk.md0p1.in‚ 208.765625 disk.nvme1n1.in‚ 850.265625# disk.nvme0n1.out‚156974.6953125 network.sent‚ 3.0868577e+07 disk.sda1.in‚0 disk.sda1.out‚0# disk./.usageGB‚2720.28470993042! disk.nvme0n1.in‚ 7682.55078125 disk.md0.out‚ 1942.3046875! disk.sda.out‚1997.82080078125 disk.rbd0.in‚0 disk.rbd0.out‚4.375 proc.cpu.threads‚17 network.recv‚ 4.605654e+06 disk.sdb1.in‚0 disk.rbd2.out‚ 4.150390625 disk.md0.in‚ 208.765625! disk.sdb.out‚1997.82080078125 disk.md0p1.out‚ 1942.3046875$ memory_percent‚92.02766418457031* proc.memory.percent‚18.171024322509766 cpu‚25.226474634791014 ‹~4¸./ _wandbruntime‚13845  _runtime‚13845l¡…ɹ.à _runtime‚13845.493949279 _step‚943 & lr-AdamW/pg1‚5.313922016932212e-06 & lr-AdamW/pg2‚5.313922016932212e-06 trainer/global_step‚11499 % _timestamp‚1.7885384619583921e+09Wlnoκ.È & lr-AdamW/pg1‚5.313922016932212e-06 & lr-AdamW/pg2‚5.313922016932212e-06 trainer/global_step‚11499 % _timestamp‚1.7885384619583921e+09 _runtime‚13845.493949279 _step‚943¯T ¹û4»./ _wandbruntime‚13845  _runtime‚13845>ÚÔt (¼.†P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚969409260048896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚969409260048896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13760737379840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291358249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚969409278051840 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213923574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚13845.495019354 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 240625684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531957589 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213924195 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 trainer/global_step‚11499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950444144612401 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13178549185024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240625686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982586899262464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚982586899262464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13177639213568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 531983935 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291358249 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1083281055909888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144176204 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13177639213568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1069520294589952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291331905 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885384619594297e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 240625686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚969409278051840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13128371101249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937315790550512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13129279076253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144176202 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982587827236864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13178549185024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640)v×(½.‹P `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937315790550512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13129279076253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144176202 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982587827236864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13178549185024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚969409260048896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚969409260048896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13760737379840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291358249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚969409278051840 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213923574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚13845.495019354 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 240625684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531957589 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213924195 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 trainer/global_step‚11499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950444144612401 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13178549185024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240625686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982586899262464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚982586899262464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13177639213568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 531983935 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291358249 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1083281055909888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144176204 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13177639213568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1069520294589952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291331905 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885384619594297e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 240625686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚969409278051840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13128371101249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456°Pé‰M4¾./ _wandbruntime‚13846  _runtime‚13846‰<<Ÿ¿.N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937394791105696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 240645846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13178760673792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213941388 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1069610544234496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982669639808512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13178760673792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚969490897137664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13179670658560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144188267 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937394808145056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚13846.444642167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532028952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532002580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69753121 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950524279480223 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11499 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291383106 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083372462602752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13761918368256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13761693793792 OADeviceStatsMonitor.on_train_batch_end/allocatedUĈ^_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1069610520294400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982670567796224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 532002580 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144188269 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1083372214088192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 240645844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13179670658560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚969490879134720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 240645844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69753744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291356736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚945 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13130396349739 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ã''À.•N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚13846.444642167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532028952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532002580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69753121 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950524279480223 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11499 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291383106 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083372462602752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13761918368256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13761693793792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1069610520294400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982670567796224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 532002580 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144188269 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1083372214088192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 240645844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13179670658560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚969490879134720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 240645844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69753744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291356736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚945 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13130396349739 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937394791105696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 240645846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13178760673792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213941388 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1069610544234496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982669639808512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13178760673792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚969490897137664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13179670658560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144188267 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937394808145056±V¿,Ø4Á./ _wandbruntime‚13846  _runtime‚13846l~» \Â.Ö 0loss/FRt1/assignment_loss‚3.3014535416256297 /loss/FRt1/detection_loss‚3.1726206568154423 .loss/FRt4/detection_loss‚2.246578005479371 _runtime‚13846.446186304 2loss/SRqqt3/assignment_loss‚0.8620973685113736 2loss/SRqqt4/assignment_loss‚1.6438646327035549 1loss/FBt2/assignment_loss‚0.13118375664195447 /loss/FRt2/detection_loss‚2.3329387890432907 0loss/SRqqt3/detection_loss‚0.849136018627646 trainer/global_step‚11499 % _timestamp‚1.7885384629109144e+09 .loss/FRt3/assignment_loss‚1.16719650213535 1loss/SRqqt1/assignment_loss‚1.235610303397362 2loss/SRqqt2/assignment_loss‚1.3807188004822186 0loss/FBt1/assignment_loss‚0.1771776683958006 0loss/SRqqt2/detection_loss‚1.740950789788495 epoch‚9 /loss/FRt2/assignment_loss‚2.680153049271682 1loss/FBt3/assignment_loss‚0.03468853070263701 /loss/FBt2/detection_loss‚2.2315068195263543 /loss/FRt3/detection_loss‚1.4723611472031244 /loss/FBt1/detection_loss‚2.1282491959058323 0loss/FBt3/detection_loss‚0.48263729400932787 .loss/FBt4/detection_loss‚3.486064067908696 2loss/FBt4/assignment_loss‚0.012105972680436545 1loss/SRqqt1/detection_loss‚2.0443019011081796 %loss/total_loss‚40.07344045444343 /loss/FRt4/assignment_loss‚3.174875908709587 0loss/SRqqt4/detection_loss‚2.084969733770077 _step‚946:Äá9aÃ.Û /loss/FRt1/detection_loss‚3.1726206568154423 .loss/FRt4/detection_loss‚2.246578005479371 _runtime‚13846.446186304 2loss/SRqqt3/assignment_loss‚0.8620973685113736 2loss/SRqqt4/assignment_loss‚1.6438646327035549 1loss/FBt2/assignment_loss‚0.13118375664195447 /loss/FRt2/detection_loss‚2.3329387890432907 0loss/SRqqt3/detection_loss‚0.849136018627646 trainer/global_step‚11499 % _timestamp‚1.7885384629109144e+09 .loss/FRt3/assignment_loss‚1.16719650213535 1loss/SRqqt1/assignment_loss‚1.235610303397362 2loss/SRqqt2/assignment_loss‚1.3807188004822186 0loss/FBt1/assignment_loss‚0.1771776683958006 0loss/SRqqt2/detection_loss‚1.740950789788495 epoch‚9 /loss/FRt2/assignment_loss‚2.680153049271682 1loss/FBt3/assignment_loss‚0.03468853070263701 /loss/FBt2/detection_loss‚2.2315068195263543 /loss/FRt3/detection_loss‚1.4723611472031244 /loss/FBt1/detection_loss‚2.1282491959058323 0loss/FBt3/detection_loss‚0.48263729400932787 .loss/FBt4/detection_loss‚3.486064067908696 2loss/FBt4/assignment_loss‚0.012105972680436545 1loss/SRqqt1/detection_loss‚2.0443019011081796 %loss/total_loss‚40.07344045444343 /loss/FRt4/assignment_loss‚3.174875908709587 0loss/SRqqt4/detection_loss‚2.084969733770077 _step‚946 0loss/FRt1/assignment_loss‚3.3014535416256297²Ä5ôœÄ.‚(:‘ ìÜëÔà¢ÓÞ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yœ}´‚Å.‚(:÷ ìÜëÔŠ¨²á disk.rbd2.in‚0 disk.sda.in‚ 133.16796875! disk.sda.out‚1999.43212890625 disk.md0p1.out‚ 1943.85546875 disk.rbd0.in‚0" disk.sda2.out‚1999.43212890625 disk.dm-0.in‚ 8532.81640625 disk.sdb2.in‚ 75.59765625) proc.memory.availableMB‚ 1305.90234375 network.recv‚ 4.609839e+06 disk.md0p1.in‚ 208.765625 disk.rbd2.out‚ 4.150390625$ disk.nvme0n1.out‚157109.47265625 disk.sda2.in‚ 133.16796875! disk.sdb.out‚1999.43212890625! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.md0.out‚ 1943.85546875 disk.dm-0.out‚222163.0546875" disk.sdb2.out‚1999.43212890625) disk./.usagePercent‚76.07110020936183! disk.nvme0n1.in‚ 7682.55078125 disk.nvme1n1.in‚ 850.265625* proc.memory.percent‚18.171024322509766 disk.rbd0.out‚4.375 disk.sdb1.out‚0 disk.md0.in‚ 208.765625# disk.nvme1n1.out‚65053.58203125 network.sent‚ 3.0887614e+07$ disk./.usageGB‚2720.671012878418 disk.sda1.out‚0 disk.sdb.in‚ 75.59765625 disk.sda1.in‚0$ memory_percent‚92.02940464019775 cpu‚25.226245537748976 disk.sdb1.in‚0“Óí”Æ.‚(:‰ ûÜëÔž¤äÞ gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ3BcmÇ.‚(:â ûÜëÔ¬©ïß network.sent‚ 3.098199e+07 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 4.150390625 disk.nvme1n1.in‚ 850.265625$ disk.nvme0n1.out‚157244.80078125 disk.sda2.in‚ 133.16796875) proc.memory.availableMB‚ 1305.90234375! proc.memory.rssMB‚ 2977.140625 disk.dm-0.in‚ 8532.81640625* proc.memory.percent‚18.171024322509766 network.recv‚ 4.616653e+06 disk.sdb1.in‚0 disk.md0.in‚ 208.765625 disk.sdb.in‚ 75.59765625 disk.rbd0.out‚4.375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.sdb.out‚2001.396484375 disk.sdb2.out‚2001.396484375% disk./.usageGB‚2720.6749534606934! disk.dm-0.out‚222298.99609375 disk.md0p1.out‚1945.75$ memory_percent‚92.02940464019775 disk.sda2.out‚2001.396484375 disk.md0.out‚1945.75 disk.sdb2.in‚ 75.59765625 disk.sda.out‚2001.396484375 cpu‚25.225989170036108" disk.nvme1n1.out‚ 65054.1953125( disk./.usagePercent‚76.0712103896915 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 proc.cpu.threads‚17¼= ”È.‚(:‰ ŠÝëÔóËÐÞ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈJž¹{É.‚(:ð ŠÝëÔô­ýß# disk.nvme0n1.out‚157314.2109375) disk./.usagePercent‚76.07120942974672 disk.sdb1.out‚0 disk.md0p1.in‚ 208.765625 network.sent‚ 3.0989937e+07 disk.sdb1.in‚0 cpu‚25.225781048029774 disk.sdb2.in‚ 75.59765625" disk.sdb2.out‚2002.92431640625 disk.nvme1n1.in‚ 850.265625 proc.cpu.threads‚17 disk.rbd2.out‚ 4.150390625 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625! disk.nvme1n1.out‚ 65054.328125! proc.memory.rssMB‚ 2977.140625 disk.rbd0.out‚4.375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚222368.5390625& proc.memory.availableMB‚ 1305.65625 network.recv‚ 4.620596e+06 disk.md0.out‚ 1947.21484375* proc.memory.percent‚18.171024322509766$ disk./.usageGB‚2720.674919128418! disk.nvme0n1.in‚ 7682.55078125 disk.rbd0.in‚0" disk.sda2.out‚2002.92431640625 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625 disk.md0p1.out‚ 1947.21484375# memory_percent‚92.0309066772461! disk.sdb.out‚2002.92431640625 disk.sda.in‚ 133.16796875! disk.sda.out‚2002.92431640625cë£ï4Ê./ _wandbruntime‚13901  _runtime‚13901nð¶òÈË. & lr-AdamW/pg1‚4.454567863481616e-06 & lr-AdamW/pg2‚4.454567863481616e-06 trainer/global_step‚11549 $ _timestamp‚1.788538517598625e+09 _runtime‚13901.134108949 _step‚947t;OmÍÌ.Ç _runtime‚13901.134108949 _step‚947 & lr-AdamW/pg1‚4.454567863481616e-06 & lr-AdamW/pg2‚4.454567863481616e-06 trainer/global_step‚11549 $ _timestamp‚1.788538517598625e+09³Y;ˆô4Í./ _wandbruntime‚13901  _runtime‚13901æ¡û'Î.…P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214817006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13185156331361 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 241633686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292576807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70036945 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚948 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1074031603752448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚986724868953088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1087850781870080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚973490214340096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986723940978688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚973490214340096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 534236837 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144779452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚941266670277712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292603151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074031579812352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788538517599417e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214816397 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13819414062592 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144779454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986723940978688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13234636610048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚941266653238352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 534210491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13234636610048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_splitJ ¸U.all.peak‚1268êñ³(Ï.ŠP _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1087851017815040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986723940978688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13234636610048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚941266653238352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 534210491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13234636610048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214817006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13185156331361 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 241633686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292576807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70036945 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚948 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1074031603752448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚986724868953088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1087850781870080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚973490214340096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986723940978688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚973490214340096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 534236837 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144779452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚941266670277712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292603151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074031579812352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788538517599417e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214816397 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13819414062592 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144779454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292628008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7885385185736423e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚986807609512448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚973571833425920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚954531036462531 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13185365629635 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1074121829456896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214834755 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074121805516800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 241653846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214834137 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 534255482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13820366031360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚941345687872256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚13902.10956707 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚973571851428864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144791519 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1087942426354176 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1087942171548160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 241653844 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 241653844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚941345670832896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13820596897280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11549 _step‚949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292601638 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 241653846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534281854 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13234848098816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 534255482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13186273604847 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292601638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 534281854 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚986807609512448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚986806681524736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13235758083584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70043236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70042620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚973571851428864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144791517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292628008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712EÖí'Ò.”N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 241653844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚941345670832896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13820596897280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11549 _step‚949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292601638 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 241653846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534281854 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13234848098816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 534255482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13186273604847 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292601638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 534281854 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚986807609512448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚986806681524736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13235758083584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70043236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70042620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚973571851428864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144791517 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292628008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292628008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7885385185736423e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚986807609512448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚973571833425920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚954531036462531 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13185365629635 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1074121829456896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214834755 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074121805516800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 241653846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214834137 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 534255482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13820366031360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚941345687872256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚13902.10956707 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚973571851428864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144791519 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1087942426354176 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1087942171548160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 241653844 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚11599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977571168631296 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145382702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13289814063616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990860982694912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚977571168631296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13289814063616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215709496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958457658577017 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70326185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242641686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293821709 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70326792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13241033586469 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990860982694912 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885385734450772e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13878043372544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958458583591381 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13290724035072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 242641686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1078542865034752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945217532965552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 293848053 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215708887 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092420932347392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚977571168631296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13289814063616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215709496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958457658577017 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70326185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242641686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293821709 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70326792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13241033586469 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990860982694912 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885385734450772e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13878043372544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958458583591381 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13290724035072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 242641686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1078542865034752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945217532965552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 293848053 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215708887 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092420932347392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚11599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977571168631296 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145382702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13289814063616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990860982694912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624¸—ë¼Û4ä./ _wandbruntime‚13957  _runtime‚13957ðßxHBå.N YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1092512097675264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242661844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215726627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13879006936064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1092512339898368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293846540 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958537793444839 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚977652787717120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 536508384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885385744344366e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 242661846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145394767 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚990943723240960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977652805720064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1078633114679296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70332477 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242661846 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚13957.970428975 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚990944651228672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536508384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚953 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13242150859955 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oVˆù Ôversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 215727246 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977652805720064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13290935523840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70331860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13241242884743 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293872910 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536534756 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536534756 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13291845508608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚958538718459411 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13291845508608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945296550560096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚990944651228672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006MÁfº'æ.•N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 242661846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145394767 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚990943723240960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977652805720064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1078633114679296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70332477 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242661846 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚13957.970428975 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚990944651228672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536508384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚953 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13242150859955 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 215727246 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977652805720064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13290935523840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70331860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13241242884743 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293872910 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536534756 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536534756 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13291845508608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚958538718459411 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13291845508608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945296550560096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚990944651228672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1092512097675264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242661844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215726627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13879006936064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1092512339898368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293846540 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958537793444839 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚977652787717120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 536508384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 % _timestamp‚1.7885385744344366e+09¹cùŽ^4ç./ _wandbruntime‚13957  _runtime‚13957ÁöZè.Ô 0loss/FRt3/assignment_loss‚1.1651592525568875 /loss/FRt1/detection_loss‚3.2738558024168016 1loss/FBt2/assignment_loss‚0.14514367721276358 /loss/FBt4/detection_loss‚2.4833177576462426 epoch‚9 /loss/FRt2/assignment_loss‚2.985028362936444 0loss/SRqqt1/detection_loss‚2.735445721582933 1loss/SRqqt3/detection_loss‚0.8413481971354417 % _timestamp‚1.7885385744362156e+09 1loss/FBt3/assignment_loss‚0.06004850342801164 0loss/SRqqt2/detection_loss‚2.167603824888506 0loss/SRqqt4/detection_loss‚2.081225982024556 trainer/global_step‚11599 _runtime‚13957.97146852 /loss/FRt1/assignment_loss‚3.538072460889816 .loss/FRt4/assignment_loss‚2.84077379728357 1loss/SRqqt3/assignment_loss‚0.892850151485412 /loss/FRt2/detection_loss‚2.7406894043620733 /loss/FRt3/detection_loss‚1.4301779859163093 .loss/FBt1/detection_loss‚2.544490784406662 &loss/total_loss‚41.982332483542564 2loss/SRqqt1/assignment_loss‚1.4132316972721706 1loss/FBt1/assignment_loss‚0.09833273247459147 0loss/FBt4/assignment_loss‚0.0811852508460967 /loss/FBt2/detection_loss‚2.3059252887964248 2loss/SRqqt4/assignment_loss‚1.3062301239087468 .loss/FRt4/detection_loss‚2.232500978279859 2loss/SRqqt2/assignment_loss‚1.9992556562177808 /loss/FBt3/detection_loss‚0.6204390895744676 _step‚954 r&_é.Ù % _timestamp‚1.7885385744362156e+09 1loss/FBt3/assignment_loss‚0.06004850342801164 0loss/SRqqt2/detection_loss‚2.167603824888506 0loss/SRqqt4/detection_loss‚2.081225982024556 trainer/global_step‚11599 _runtime‚13957.97146852 /loss/FRt1/assignment_loss‚3.538072460889816 .loss/FRt4/assignment_loss‚2.84077379728357 1loss/SRqqt3/assignment_loss‚0.892850151485412 /loss/FRt2/detection_loss‚2.7406894043620733 /loss/FRt3/detection_loss‚1.4301779859163093 .loss/FBt1/detection_loss‚2.544490784406662 &loss/total_loss‚41.982332483542564 2loss/SRqqt1/assignment_loss‚1.4132316972721706 1loss/FBt1/assignment_loss‚0.09833273247459147 0loss/FBt4/assignment_loss‚0.0811852508460967 /loss/FBt2/detection_loss‚2.3059252887964248 2loss/SRqqt4/assignment_loss‚1.3062301239087468 .loss/FRt4/detection_loss‚2.232500978279859 2loss/SRqqt2/assignment_loss‚1.9992556562177808 /loss/FBt3/detection_loss‚0.6204390895744676 _step‚954 0loss/FRt3/assignment_loss‚1.1651592525568875 /loss/FRt1/detection_loss‚3.2738558024168016 1loss/FBt2/assignment_loss‚0.14514367721276358 /loss/FBt4/detection_loss‚2.4833177576462426 epoch‚9 /loss/FRt2/assignment_loss‚2.985028362936444 0loss/SRqqt1/detection_loss‚2.735445721582933 1loss/SRqqt3/detection_loss‚0.8413481971354417º¾që “ê.‚(:ˆ ÕÝëÔ¶ÚÐß gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓúÄ€ë.‚(:õ ÕÝëÔªÆà network.recv‚ 4.644168e+06) disk./.usagePercent‚76.08440365755177 disk.rbd2.out‚ 4.150390625 disk.dm-0.in‚ 8532.81640625 disk.dm-0.out‚ 223171.296875! disk.sda.out‚2010.92529296875 disk.sda2.in‚ 133.16796875 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625! proc.memory.rssMB‚ 2977.140625 disk.sda1.out‚0% disk./.usageGB‚2721.1468086242676 disk.sda1.in‚0# disk.nvme1n1.out‚65058.55078125 disk.md0p1.out‚ 1954.90234375) proc.memory.availableMB‚ 1305.80859375 proc.cpu.threads‚17! disk.sdb.out‚2010.92529296875 disk.rbd0.out‚4.375# memory_percent‚92.0299768447876 network.sent‚ 3.111842e+07 disk.rbd2.in‚0 disk.md0.out‚ 1954.90234375* proc.memory.percent‚18.171024322509766 cpu‚25.224748325740073$ disk.nvme0n1.out‚158112.74609375" disk.sda2.out‚2010.92529296875 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625 disk.sdb1.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.md0.in‚ 208.765625" disk.sdb2.out‚2010.92529296875 disk.rbd0.in‚0 disk.sdb1.out‚0¾g6F“ì.‚(:ˆ äÝëÔŒ‡›ß gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!=ê.tí.‚(:é äÝëÔ“Ññà) disk./.usagePercent‚76.08276257860454 disk.sda1.in‚0 disk.md0.in‚ 208.765625* proc.memory.percent‚18.171024322509766 cpu‚25.22454084871184 disk.rbd2.out‚ 4.150390625" disk.nvme0n1.out‚ 158217.046875 disk.sdb2.in‚ 75.59765625 disk.dm-0.out‚ 223275.78125 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚4.375 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚92.03002452850342) proc.memory.availableMB‚ 1305.80078125 disk.rbd2.in‚0 disk.md0.out‚ 1956.98046875 disk.md0p1.out‚ 1956.98046875 disk.sda2.out‚2013.064453125 network.sent‚ 3.1228608e+07 disk.sda1.out‚0! disk.nvme0n1.in‚ 7682.55078125 disk.dm-0.in‚ 8532.81640625 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.265625! disk.nvme1n1.out‚ 65058.734375 network.recv‚ 4.651383e+06% disk./.usageGB‚2721.0881156921387 disk.sda2.in‚ 133.16796875 disk.sdb.out‚2013.064453125 disk.sda.out‚2013.064453125! proc.memory.rssMB‚ 2977.140625 proc.cpu.threads‚17 disk.sdb2.out‚2013.064453125 disk.rbd0.in‚0™Í§©”î.‚(:‰ óÝëÔÒÕÉà gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ápÕbqï.‚(:æ óÝëÔ—¦¡á disk.md0.in‚ 208.765625 disk.dm-0.out‚ 223569.875 disk.sdb.out‚ 2014.62109375 disk.sda.out‚ 2014.62109375$ memory_percent‚92.02845096588135* proc.memory.percent‚18.171024322509766 disk.nvme1n1.in‚ 850.265625# disk.nvme1n1.out‚65145.19140625 disk.sda1.in‚0 disk.dm-0.in‚ 8532.81640625 disk.rbd0.in‚0 cpu‚25.224334815569676$ disk./.usageGB‚2721.172119140625 disk.sdb2.in‚ 75.59765625 disk.rbd0.out‚4.375 disk.sdb1.in‚0 disk.sda2.out‚ 2014.62109375 network.sent‚ 3.1231836e+07 disk.md0.out‚ 1958.47265625 disk.sda.in‚ 133.16796875 disk.md0p1.out‚ 1958.47265625 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚ 4.150390625 disk.sda2.in‚ 133.16796875( disk./.usagePercent‚76.0851113501824 disk.sdb2.out‚ 2014.62109375 disk.md0p1.in‚ 208.765625) proc.memory.availableMB‚ 1306.05859375 disk.sdb.in‚ 75.59765625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7682.55078125$ disk.nvme0n1.out‚158424.68359375! proc.memory.rssMB‚ 2977.140625 network.recv‚ 4.655365e+06–l|”ð.‚(:‰ ‚ÞëÔœÅÏà gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X„>Oyñ.‚(:î ‚ÞëÔ¾úà disk.sdb.in‚ 75.59765625# disk.nvme1n1.out‚65148.05078125 network.sent‚ 3.1237443e+07$ disk./.usageGB‚2721.174644470215! disk.nvme0n1.in‚ 7682.55078125! disk.dm-0.out‚223676.38671875$ memory_percent‚92.03150272369385 disk.md0.in‚ 208.765625! proc.memory.rssMB‚ 2977.140625 disk.sda2.out‚2015.927734375 disk.dm-0.in‚ 8532.81640625 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1959.72265625 disk.rbd0.out‚4.375 disk.sdb1.in‚0) proc.memory.availableMB‚ 1305.55859375 disk.rbd2.in‚0 disk.md0.out‚ 1959.72265625 disk.sdb.out‚2015.927734375 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625 proc.cpu.threads‚17) disk./.usagePercent‚76.08518195945466 disk.rbd2.out‚ 4.150390625# disk.nvme0n1.out‚158528.3359375 disk.sda.in‚ 133.16796875 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 cpu‚25.22414964894483 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 disk.sdb2.out‚2015.927734375 network.recv‚ 4.658918e+06 disk.sda.out‚2015.927734375 disk.rbd0.in‚0ÐÂÐ 4ò./ _wandbruntime‚14013  _runtime‚14013ÄP_Êó.Ä ' lr-AdamW/pg2‚2.9619968419524526e-06 trainer/global_step‚11649 % _timestamp‚1.7885386301560955e+09 _runtime‚14013.69166974 _step‚955 ' lr-AdamW/pg1‚2.9619968419524526e-06ð²[ÍÏô.É ' lr-AdamW/pg1‚2.9619968419524526e-06 ' lr-AdamW/pg2‚2.9619968419524526e-06 trainer/global_step‚11649 % _timestamp‚1.7885386301560955e+09 _runtime‚14013.69166974 _step‚955»êû4õ./ _wandbruntime‚14013  _runtime‚14013¶ˆyö.†P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚11649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13296910841577 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13345901488640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216601406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚949168429732112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885386301570947e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538742641 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083054174197248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981652140925440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 243649686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 295092955 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216600778 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13346811460096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 295066611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538716295 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 145985954 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 243649684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994998952385536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096990972730880 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083054150257152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.onÝwù¼“_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _runtime‚14013.692729585 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 295066611 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 295092955 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1096990707425792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 _step‚956 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994998024411136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 243649686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994998024411136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981652122922496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 243649684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70615452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚962465340573689 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994998024411136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981652122922496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 243649684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70615452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚962465340573689 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13296910841577 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13345901488640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216601406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚949168429732112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885386301570947e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538742641 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083054174197248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981652140925440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 243649686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 295092955 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216600778 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13346811460096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 295066611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538716295 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 145985954 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 243649684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994998952385536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096990972730880 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083054150257152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249¼‘³U4ø./ _wandbruntime‚14014  _runtime‚140148S:'ù.N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995081692944896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚962545475441719 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243669846 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295117812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13347022948864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚949247447326656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 243669844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13347932933632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13937751752704 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13347932933632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295091442 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1097082367840256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚595 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7885386312466793e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538787658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981733742008320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083144375961600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538761286 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 243669844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13937967938560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538761286 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13298028115063 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 295091442 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13347022948864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216618503 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1097082127714304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295117812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _runtime‚14014.783543684 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538787658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70621079 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083144399901696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243669846 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„6u'ú.•N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243669846 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295117812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13347022948864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚949247447326656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 243669844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13347932933632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13937751752704 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13347932933632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295091442 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1097082367840256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚595 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7885386312466793e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538787658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981733742008320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083144375961600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538761286 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 243669844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13937967938560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538761286 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13298028115063 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 295091442 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13347022948864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216618503 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1097082127714304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295117812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _runtime‚14014.783543684 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538787658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70621079 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083144399901696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243669846 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995081692944896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚962545475441719 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185½e¡/Ð4û./ _wandbruntime‚14014  _runtime‚14014ÁŽ^’ü.Ñ .loss/FRt1/assignment_loss‚3.40793319309459 /loss/FRt2/assignment_loss‚3.345802716082997 .loss/FRt2/detection_loss‚2.2899193612ói1ŸÅ39299 &loss/total_loss‚41.938294578679375 0loss/FRt4/assignment_loss‚3.3938501803647902 2loss/SRqqt3/assignment_loss‚0.8312057094763587 2loss/SRqqt4/assignment_loss‚1.2685625032068064 1loss/FBt2/assignment_loss‚0.17301496863365085 0loss/FBt4/assignment_loss‚0.5647994930817125 /loss/FRt4/detection_loss‚2.4702463448934613 .loss/FBt2/detection_loss‚2.631525419652462 0loss/FRt3/assignment_loss‚1.0753706213500764 2loss/SRqqt1/assignment_loss‚2.4232260517776014 .loss/FRt1/detection_loss‚2.764917889062096 1loss/FBt1/assignment_loss‚0.13989600674093547 /loss/FRt3/detection_loss‚1.1489578295602567 -loss/FBt3/detection_loss‚0.55433185160443 # _timestamp‚1.78853863124924e+09 _step‚958 /loss/FBt4/detection_loss‚2.1958770950635276 1loss/SRqqt1/detection_loss‚2.8224904596805573 0loss/SRqqt2/detection_loss‚2.015760900009246 2loss/SRqqt2/assignment_loss‚1.2912028850100579 1loss/FBt3/assignment_loss‚0.05488514059659424 0loss/SRqqt3/detection_loss‚0.800582595441616 epoch‚9 0loss/SRqqt4/detection_loss‚2.023203677473924 .loss/FBt1/detection_loss‚2.250731685582329 trainer/global_step‚11649 _runtime‚14014.784514159ÞýK\ý.Ö 0loss/FRt3/assignment_loss‚1.0753706213500764 2loss/SRqqt1/assignment_loss‚2.4232260517776014 .loss/FRt1/detection_loss‚2.764917889062096 1loss/FBt1/assignment_loss‚0.13989600674093547 /loss/FRt3/detection_loss‚1.1489578295602567 -loss/FBt3/detection_loss‚0.55433185160443 # _timestamp‚1.78853863124924e+09 _step‚958 /loss/FBt4/detection_loss‚2.1958770950635276 1loss/SRqqt1/detection_loss‚2.8224904596805573 0loss/SRqqt2/detection_loss‚2.015760900009246 2loss/SRqqt2/assignment_loss‚1.2912028850100579 1loss/FBt3/assignment_loss‚0.05488514059659424 0loss/SRqqt3/detection_loss‚0.800582595441616 epoch‚9 0loss/SRqqt4/detection_loss‚2.023203677473924 .loss/FBt1/detection_loss‚2.250731685582329 trainer/global_step‚11649 _runtime‚14014.784514159 .loss/FRt1/assignment_loss‚3.40793319309459 /loss/FRt2/assignment_loss‚3.345802716082997 .loss/FRt2/detection_loss‚2.289919361239299 &loss/total_loss‚41.938294578679375 0loss/FRt4/assignment_loss‚3.3938501803647902 2loss/SRqqt3/assignment_loss‚0.8312057094763587 2loss/SRqqt4/assignment_loss‚1.2685625032068064 1loss/FBt2/assignment_loss‚0.17301496863365085 0loss/FBt4/assignment_loss‚0.5647994930817125 /loss/FRt4/detection_loss‚2.4702463448934613 .loss/FBt2/detection_loss‚2.631525419652462¾<\k”þ.‚(:‰ ‘ÞëÔ·ÛÉá gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yi=ðxÿ.‚(:í ‘ÞëÔâ% disk./.usageGB‚2721.2995262145996 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda.in‚ 133.16796875! disk.nvme1n1.out‚ 65213.484375 disk.rbd0.out‚4.375 disk.md0.out‚ 1961.17578125 disk.dm-0.out‚ 223845.71875! disk.sdb2.out‚2017.4345703125 disk.sda.out‚2017.4345703125 disk.md0p1.out‚ 1961.17578125 disk.rbd0.in‚0" disk.nvme0n1.out‚ 158632.234375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb.out‚2017.4345703125! disk.nvme0n1.in‚ 7682.55078125 disk.nvme1n1.in‚ 850.265625) proc.memory.availableMB‚ 1305.69140625 proc.cpu.threads‚17 network.recv‚ 4.663479e+06# memory_percent‚92.0306921005249 cpu‚25.223940475991743 network.sent‚ 3.1256541e+07 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625 disk.sdb.in‚ 75.59765625* proc.memory.percent‚18.171024322509766 disk.rbd2.out‚ 4.150390625) disk./.usagePercent‚76.08867370529472 disk.sdb2.in‚ 75.59765625 disk.sda2.in‚ 133.16796875! disk.sda2.out‚2017.4345703125 disk.md0p1.in‚ 208.765625! proc.memory.rssMB‚ 2977.140625ÞÅIJ”€/‚(:‰  ÞëÔΛ¾à gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥n/‚(:ã  ÞëÔüÓõà disk.md0.out‚ 1963.30859375 disk.sdb1.out‚0! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.09218241015941 disk.sda2.in‚ 133.16796875 disk.sda2.out‚ 2019.640625 disk.dm-0.in‚ 8532.81640625! disk.dm-0.out‚224014.58984375# disk.nvme1n1.out‚65278.69921875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625 disk.sdb.in‚ 75.59765625 disk.sdb2.out‚ 2019.640625 disk.rbd0.in‚0 cpu‚25.223724964400486 network.sent‚ 3.1351733e+07% disk./.usageGB‚2721.4250144958496 disk.sdb.out‚ 2019.640625 disk.nvme1n1.in‚ 850.265625) proc.memory.availableMB‚ 1305.44921875 network.recv‚ 4.670669e+06! disk.nvme0n1.in‚ 7682.55078125 disk.sdb1.in‚0$ memory_percent‚92.03217029571533" disk.nvme0n1.out‚ 158735.890625 disk.sda.in‚ 133.16796875 disk.md0p1.in‚ 208.765625 disk.md0p1.out‚ 1963.30859375 disk.rbd0.out‚4.375 disk.sda1.out‚0* proc.memory.percent‚18.171024322509766 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 2019.640625Yöú”‚/‚(:‰ ¯ÞëÔôáÕà gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Šofƒ/‚(:Û ¯ÞëԤܙâ disk.md0.out‚ 1965.03515625 disk.sdb2.out‚ 2021.4375 disk.sda.in‚ 133.16796875! disk.nvme0n1.in‚ 7682.55078125 disk.sda2.out‚ 2021.4375 disk.sdb.in‚ 75.59765625 disk.md0p1.out‚ 1965.03515625 disk.sdb1.in‚0 network.sent‚ 3.1359418e+07) disk./.usagePercent‚76.09218081025143 disk.sdb.out‚ 2021.4375 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625* proc.memory.percent‚18.171024322509766% disk./.usageGB‚2721.4249572753906! disk.dm-0.out‚224306.62890625 disk.rbd0.in‚0 disk.rbd0.out‚4.375 proc.cpu.threads‚17 disk.rbd2.out‚ 4.150390625# disk.nvme0n1.out‚158932.5859375) proc.memory.availableMB‚ 1305.69921875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.sda.out‚ 2021.4375# disk.nvme1n1.out‚65375.04296875 cpu‚25.223535152092637 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625 disk.nvme1n1.in‚ 850.265625 disk.sdb1.out‚0$ memory_percent‚92.03064441680908! proc.memory.rssMB‚ 2977.140625 network.recv‚ 4.67448e+06#<ž„/‚(:“ ¾ÞëԿȺâ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ,Æ‚…/‚(:÷ ¾ÞëÔýÅ£ã cpu‚25.223328925117006 proc.cpu.threads‚17 disk.dm-0.in‚ 8532.81640625 disk.dm-0.out‚224436.5703125 disk.md0p1.in‚ 208.765625 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1966.42578125 disk.rbd2.out‚ 4.150390625) proc.memory.availableMB‚ 1305.19921875 disk.sda1.out‚0! disk.nvme0n1.in‚ 7682.55078125 disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875$ disk./.usageGB‚2721.673366546631 disk.nvme1n1.in‚ 850.265625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 75.59765625! disk.sda.out‚2022.90087890625 network.recv‚ 4.678242e+06) disk./.usagePercent‚76.09912643744265 disk.rbd0.in‚0 disk.md0.in‚ 208.765625$ memory_percent‚92.03369617462158 disk.md0p1.out‚ 1966.42578125 network.sent‚ 3.1362915e+07 disk.rbd0.out‚4.375 disk.sda1.in‚0" disk.sda2.out‚2022.90087890625! proc.memory.rssMB‚ 2977.140625* proc.memory.percent‚18.171024322509766" disk.sdb2.out‚2022.90087890625# disk.nvme1n1.out‚65409.29296875$ disk.nvme0n1.out‚159027.27734375! disk.sdb.out‚2022.90087890625Q2ˆ¿4†// _wandbruntime‚14070  _runtime‚14070/Š@Àˇ/Å ' lr-AdamW/pg2‚2.3290910234375522e-06 trainer/global_step‚11699 % _timestamp‚1.7885386864790044e+09 _runtime‚14070.015168099 _step‚959 ' lr-AdamW/pg1‚2.3290910234375522e-06œÆ+àЈ/Ê % _timestamp‚1.7885386864790044e+09 _runtime‚14070.015168099 _step‚959 ' lr-AdamW/pg1‚2.3290910234375522e-06 ' lr-AdamW/pg2‚2.3290910234375522e-06 trainer/global_step‚11699¿¶aáC4‰// _wandbruntime‚14070  _runtime‚140708Qܾ (Š/†P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚953119292419952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 244657684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1087565435479552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 296311513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 217493872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1101561000715776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 244657684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296311513 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚11699 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1087565459419648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚966472097555997 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296337857 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚999135066127360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70904060 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13402898885120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚966471172541633 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885386864810507e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚14070.016818967 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13401988913664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999135066127360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244657686 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13995541296128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13402898885120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296337857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13351880121681 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 540969197 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚999135994101760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚953119309459312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 146589202 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244657686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚985733095216640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚985733077213696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13352788096685 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚985733095216640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217493262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70904668 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13995320902656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985733077213696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13401988913664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚14070.016818967 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13401988913664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999135066127360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244657686 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13995541296128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13402898885120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296337857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13351880121681 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 540969197 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚999135994101760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚953119309459312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 146589202 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244657686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚985733095216640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚985733077213696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13352788096685 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚985733095216640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217493262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70904668 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13995320902656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985733077213696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13401988913664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Àããÿ4Œ// _wandbruntime‚14071  _runtime‚14071ùÁ—2å /N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541040560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296362714 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1101652137309184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217511747 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1101652398406656 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087655661184000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚14071.042297918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚985814714302464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚953198327053856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788538687506002e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13996476125184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚953198310014496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541014188 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚11699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 244677846 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 244677844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13403110373888 _step‚961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13404020358656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 244677846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1087655685124096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 244677844 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146601267 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217511126 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13352997394959 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚985814696299520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13403110373888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚999218734661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13404020358656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70909859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70910478 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1087655685124096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 244677844 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146601267 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217511126 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13352997394959 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚985814696299520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13403110373888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚999218734661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13404020358656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70909859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70910478 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541040560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296362714 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1101652137309184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217511747 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1101652398406656 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087655661184000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚14071.042297918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚985814714302464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚953198327053856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788538687506002e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13996476125184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚953198310014496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541014188 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚11699Áñ)z4// _wandbruntime‚14071  _runtime‚14071øÙ]/× 2loss/SRqqt4/assignment_loss‚1.5074362735729665 %loss/total_loss‚40.03532357394141 epoch‚9 _step‚962 /loss/FRt3/assignment_loss‚1.083694226706206 /loss/FRt2/detection_loss‚2.7564334207645866 1loss/SRqqt4/detection_loss‚1.8432409353554249 % _timestamp‚1.7885386875080469e+09 1loss/FBt3/assignment_loss‚0.04168948768439492 0loss/FBt4/assignment_loss‚0.3003616971142483 /loss/FRt3/detection_loss‚1.4015061970132945 /loss/FBt2/detection_loss‚2.6690945598212155 /loss/FBt4/detection_loss‚2.1191306824867544 _runtime‚14071.043816615 1loss/SRqqt1/detection_loss‚2.3849432853360972 .loss/FBt1/detection_loss‚2.756643123924732 2loss/SRqqt2/assignment_loss‚1.4864989853110806 2loss/SRqqt3/assignment_loss‚0.9704109182274168 1loss/SRqqt2/detection_loss‚1.8911733904789234 1loss/FBt1/assignment_loss‚0.24002649385289487 .loss/FRt1/detection_loss‚2.654962965183788 0loss/SRqqt3/detection_loss‚0.933629897660873 0loss/FBt3/detection_loss‚0.44223765693604944 .loss/FRt1/assignment_loss‚3.19506213400099 2loss/SRqqt1/assignment_loss‚1.6205377527318585 1loss/FBt2/assignment_loss‚0.23256491619184191 /loss/FRt4/detection_loss‚1.9522692000582105 trainer/global_step‚11699 /loss/FRt2/assignment_loss‚3.102970201235551 /loss/FRt4/assignment_loss‚2.448805172292013¶ú¥b‘/Ü _step‚962 /loss/FRt3/assignment_loss‚1.083694226706206 /loss/FRt2/detection_loss‚2.7564334207645866 1loss/SRqqt4/detection_loss‚1.8432409353554249 % _timestamp‚1.7885386875080469e+09 1loss/FBt3/assignment_loss‚0.04168948768439492 0loss/FBt4/assignment_loss‚0.3003616971142483 /loss/FRt3/detection_loss‚1.4015061970132945 /loss/FBt2/detection_loss‚2.6690945598212155 /loss/FBt4/detection_loss‚2.1191306824867544 _runtime‚14071.043816615 1loss/SRqqt1/detection_loss‚2.3849432853360972 .loss/FBt1/detection_loss‚2.756643123924732 2loss/SRqqt2/assignment_loss‚1.4864989853110806 2loss/SRqqt3/assignment_loss‚0.9704109182274168 1loss/SRqqt2/detection_loss‚1.8911733904789234 1loss/FBt1/assignment_loss‚0.24002649385289487 .loss/FRt1/detection_loss‚2.654962965183788 0loss/SRqqt3/detection_loss‚0.933629897660873 0loss/FBt3/detection_loss‚0.44223765693604944 .loss/FRt1/assignment_loss‚3.19506213400099 2loss/SRqqt1/assignment_loss‚1.6205377527318585 1loss/FBt2/assignment_loss‚0.23256491619184191 /loss/FRt4/detection_loss‚1.9522692000582105 trainer/global_step‚11699 /loss/FRt2/assignment_loss‚3.102970201235551 /loss/FRt4/assignment_loss‚2.448805172292013 2loss/SRqqt4/assignment_loss‚1.5074362735729665 %loss/total_loss‚40.03532357394141 epoch‚9ÂÑKX\”’/‚(:‰ ÍÞëÔ‹õ¨à gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘j+“/‚(:ô ÍÞëÔÝ‹ ã disk.rbd2.in‚0" disk.sda2.out‚2024.74951171875 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875! disk.dm-0.out‚224638.77734375! disk.sdb.out‚2024.74951171875 disk.nvme1n1.in‚ 850.265625 disk.md0p1.in‚ 208.765625 disk.sdb.in‚ 75.59765625 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 disk.sda.in‚ 133.16796875! disk.sda.out‚2024.74951171875 disk.md0p1.out‚ 1968.21484375 disk.rbd0.in‚0 disk.rbd0.out‚4.375 network.sent‚ 3.1382007e+07$ disk./.usageGB‚2721.675319671631 disk.rbd2.out‚ 4.150390625" disk.sdb2.out‚2024.74951171875) proc.memory.availableMB‚ 1305.66796875! proc.memory.rssMB‚ 2977.140625 cpu‚25.22312674913922( disk./.usagePercent‚76.0991810476351! disk.nvme0n1.in‚ 7682.55078125! disk.nvme1n1.out‚ 65539.828125 disk.sdb1.in‚0 network.recv‚ 4.682987e+06 disk.sda1.out‚0$ disk.nvme0n1.out‚159098.94921875 disk.md0.out‚ 1968.21484375$ memory_percent‚92.03083515167236 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625 disk.sdb2.in‚ 75.59765625 disk.sda1.in‚0Ö’”/‚(:‡ ÜÞëÔ¼àÓà gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.352' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ÑŽz•/‚(:ï ÜÞëÔÎÒ“â! disk.sda2.out‚2026.3681640625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda.out‚2026.3681640625# disk.nvme1n1.out‚65605.07421875$ disk./.usageGB‚2721.675319671631$ memory_percent‚92.03228950500488( disk./.usagePercent‚76.0991810476351 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.md0p1.in‚ 208.765625! proc.memory.rssMB‚ 2977.140625 disk.dm-0.in‚ 8532.81640625! disk.sdb2.out‚2026.3681640625( proc.memory.availableMB‚ 1305.4296875 proc.cpu.threads‚17 disk.md0.out‚ 1969.76953125! disk.dm-0.out‚224883.83984375 disk.sdb.in‚ 75.59765625 cpu‚25.22292928325909 disk.sdb1.in‚0! disk.nvme0n1.in‚ 7682.55078125" disk.nvme0n1.out‚ 159278.765625 disk.sda2.in‚ 133.16796875 disk.sdb.out‚2026.3681640625 disk.sdb2.in‚ 75.59765625 disk.nvme1n1.in‚ 850.265625 disk.rbd0.in‚0 disk.rbd0.out‚4.375 disk.rbd2.out‚ 4.150390625 disk.sda.in‚ 133.16796875 disk.md0p1.out‚ 1969.76953125* proc.memory.percent‚18.171024322509766 network.sent‚ 3.1479782e+07 network.recv‚ 4.690747e+06 disk.sda1.in‚0F&(á”–/‚(:‰ ëÞëÔ¸˜¬à gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ÙPo{—/‚(:ð ëÞëÔ¢¥ýà" disk.sdb2.out‚2027.78955078125 disk.sda.in‚ 133.16796875 disk.rbd0.out‚4.375 cpu‚25.22275015643042( disk./.usagePercent‚76.0991867006433 disk.sda1.in‚0 disk.sda2.in‚ 133.16796875" disk.sda2.out‚2027.78955078125 disk.md0.out‚ 1971.13671875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17$ disk./.usageGB‚2721.675521850586# disk.nvme1n1.out‚65670.95703125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1305.41796875 network.recv‚ 4.694571e+06 disk.sdb2.in‚ 75.59765625 disk.sdb1.out‚0 disk.rbd2.out‚ 4.150390625! disk.nvme0n1.in‚ 7682.55078125 disk.rbd0.in‚0 disk.md0.in‚ 208.765625 disk.dm-0.in‚ 8532.81640625! disk.dm-0.out‚224977.70703125$ memory_percent‚92.03236103057861 network.sent‚ 3.1485227e+07! disk.sda.out‚2027.78955078125 disk.nvme0n1.out‚ 159306.75! proc.memory.rssMB‚ 2977.140625 disk.nvme1n1.in‚ 850.265625 disk.md0p1.out‚ 1971.13671875 disk.sda1.out‚0 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0 disk.sdb.in‚ 75.59765625! disk.sdb.out‚2027.78955078125K°:/4˜// _wandbruntime‚14126  _runtime‚14126ÇôëË™/Å ' lr-AdamW/pg1‚1.7719527827086063e-06 ' lr-AdamW/pg2‚1.7719527827086063e-06 trainer/global_step‚11749 % _timestamp‚1.7885387432291737e+09 _runtime‚14126.764730611 _step‚963ÿÌž3К/Ê % _timestamp‚1.7885387432291737e+09 _runtime‚14126.764730611 _step‚963 ' lr-AdamW/pg1‚1.7719527827086063e-06 ' lr-AdamW/pg2‚1.7719527827086063e-06 trainer/global_step‚11749ÿ¢ª4›// _wandbruntime‚14126  _runtime‚14126;:3ù"œ/ŠP 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957070172147152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚11749 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218385564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚14126.766266958 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003273035817984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13458986310144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245665684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989814031504896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 245665686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚989814049507840 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147192452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13408665351793 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚989814031504896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13407757376789 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147192454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1106130563334144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218386203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71193112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚957070189186512 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 543222099 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245665686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245665684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71193749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003272107843584 % _timestamp‚1.7885387432305872e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1106130826542080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1092076720701952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14054081900032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543248445 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13458076338688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1003273035817984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13458986310144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297556415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14053842632192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Èåã»(/P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218386203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71193112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚957070189186512 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 543222099 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245665686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245665684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71193749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003272107843584 % _timestamp‚1.7885387432305872e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1106130826542080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1092076720701952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14054081900032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543248445 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13458076338688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1003273035817984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13458986310144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297556415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14053842632192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957070172147152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚11749 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218385564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚14126.766266958 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003273035817984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13458986310144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245665684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989814031504896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 245665686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚989814049507840 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147192452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13408665351793 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚989814031504896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13407757376789 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147192454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1106130563334144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Äê ò4ž// _wandbruntime‚14127  _runtime‚14127ãýµš'Ÿ/”N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 245685846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 297581246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚970558064391763 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚957149189741696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13409782625279 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚989895668593664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71198767 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11749 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚989895650590720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 297581246 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106221946783232 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1092166970346496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1003354848389632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 245685846 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13459197798912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543293462 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 543267090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 245685844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚989895668593664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14055000376832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚957149206781056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13408874650067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543293462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚14127.846992892 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218403284 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14055235437056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚989895650590720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13460107783680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218403902 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 % _timestamp‚1.7885387443108444e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 297607616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13459197798912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1092166946406400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚970558989406335 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0NÔËŒ% /™N WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚970558989406335 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 245685846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 297581246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚970558064391763 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚957149189741696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13409782625279 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚989895668593664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71198767 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11749 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚989895650590720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 297581246 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106221946783232 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1092166970346496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1003354848389632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 245685846 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13459197798912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543293462 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 543267090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 245685844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚989895668593664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14055000376832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚957149206781056 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13408874650067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543293462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚14127.846992892 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218403284 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14055235437056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚989895650590720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13460107783680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218403902 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 % _timestamp‚1.7885387443108444e+09 OADeviceStatsMonitor.oëÅ“n_train_batch_end/active.small_pool.allocated‚ 297607616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13459197798912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1092166946406400Åq‡Ô4¡// _wandbruntime‚14127  _runtime‚14127}5æ`¢/Ú 2loss/SRqqt4/assignment_loss‚1.7210769071465446 1loss/FBt2/assignment_loss‚0.14542487213839195 /loss/FRt1/detection_loss‚2.8759502291679384 1loss/SRqqt1/detection_loss‚2.1817984521389007 1loss/SRqqt2/detection_loss‚2.4493459264437356 /loss/FRt4/assignment_loss‚3.316078369523965 epoch‚9 % _timestamp‚1.7885387443127031e+09 2loss/SRqqt2/assignment_loss‚1.5875130450284038 1loss/SRqqt3/detection_loss‚0.8137656577715748 trainer/global_step‚11749 0loss/FRt1/assignment_loss‚2.8339198136329653 /loss/FRt3/assignment_loss‚1.161394558408681 2loss/SRqqt1/assignment_loss‚1.9943457668026288 0loss/FBt1/assignment_loss‚0.2849872889680889 /loss/FBt4/assignment_loss‚0.411179450437567 /loss/FRt3/detection_loss‚1.4275536549770658 /loss/FRt4/detection_loss‚2.3598639932917616 &loss/total_loss‚40.442147058834664 /loss/FRt2/assignment_loss‚3.372073871748788 2loss/FBt3/assignment_loss‚0.007972858096651072 1loss/SRqqt4/detection_loss‚1.9666696921700524 .loss/FBt1/detection_loss‚1.941461812366139 /loss/FBt3/detection_loss‚0.5482391099065123 _runtime‚14127.847972937 2loss/SRqqt3/assignment_loss‚0.9721632423271475 /loss/FRt2/detection_loss‚2.4007517270608028 /loss/FBt2/detection_loss‚1.9953978806734085 /loss/FBt4/detection_loss‚1.6732188786069553 _step‚966[<«öe£/ß 1loss/FBt2/assignment_loss‚0.14542487213839195 /loss/FRt1/detection_loss‚2.8759502291679384 1loss/SRqqt1/detection_loss‚2.1817984521389007 1loss/SRqqt2/detection_loss‚2.4493459264437356 /loss/FRt4/assignment_loss‚3.316078369523965 epoch‚9 % _timestamp‚1.7885387443127031e+09 2loss/SRqqt2/assignment_loss‚1.5875130450284038 1loss/SRqqt3/detection_loss‚0.8137656577715748 trainer/global_step‚11749 0loss/FRt1/assignment_loss‚2.8339198136329653 /loss/FRt3/assignment_loss‚1.161394558408681 2loss/SRqqt1/assignment_loss‚1.9943457668026288 0loss/FBt1/assignment_loss‚0.2849872889680889 /loss/FBt4/assignment_loss‚0.411179450437567 /loss/FRt3/detection_loss‚1.4275536549770658 /loss/FRt4/detection_loss‚2.3598639932917616 &loss/total_loss‚40.442147058834664 /loss/FRt2/assignment_loss‚3.372073871748788 2loss/FBt3/assignment_loss‚0.007972858096651072 1loss/SRqqt4/detection_loss‚1.9666696921700524 .loss/FBt1/detection_loss‚1.941461812366139 /loss/FBt3/detection_loss‚0.5482391099065123 _runtime‚14127.847972937 2loss/SRqqt3/assignment_loss‚0.9721632423271475 /loss/FRt2/detection_loss‚2.4007517270608028 /loss/FBt2/detection_loss‚1.9953978806734085 /loss/FBt4/detection_loss‚1.6732188786069553 _step‚966 2loss/SRqqt4/assignment_loss‚1.7210769071465446Æ3•/N”¤/‚(:‰ úÞëÔŠ²ùß gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Œ]€¥/‚(:õ úÞëÔ÷þÉà network.sent‚ 3.1488623e+07 disk.sda2.in‚ 133.16796875" disk.sda2.out‚2029.19970703125 disk.sda.in‚ 133.16796875) proc.memory.availableMB‚ 1305.11328125* proc.memory.percent‚18.171024322509766 disk.nvme1n1.in‚ 850.265625 disk.rbd2.out‚ 4.150390625! disk.sdb.out‚2029.19970703125! disk.sda.out‚2029.19970703125 disk.md0p1.out‚ 1972.48828125 disk.rbd0.in‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚159462.23046875 disk.md0.out‚ 1972.48828125 disk.sdb.in‚ 75.59765625 disk.md0.in‚ 208.765625 disk.dm-0.out‚225198.4140625 disk.sdb2.in‚ 75.59765625# memory_percent‚92.0342206954956! proc.memory.rssMB‚ 2977.140625 disk.md0p1.in‚ 208.765625! disk.nvme0n1.in‚ 7682.55078125 disk.dm-0.in‚ 8532.81640625 cpu‚25.22253331802141 network.recv‚ 4.69837e+06 disk.sdb1.in‚0 disk.sda1.in‚0" disk.sdb2.out‚2029.19970703125 proc.cpu.threads‚17) disk./.usagePercent‚76.09918894051448 disk.sdb1.out‚0 disk.sda1.out‚0% disk./.usageGB‚2721.6756019592285# disk.nvme1n1.out‚65736.18359375 disk.rbd0.out‚4.375¿Oû”¦/‚(:‰ ‰ßëÔ º¦â gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MƒYMt§/‚(:é ‰ßëÔú‡¦å disk.md0p1.in‚ 208.765625 disk.rbd0.out‚4.375 disk.rbd2.in‚0 network.sent‚ 3.1596834e+07 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.sdb.out‚ 2031.08203125* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.nvme1n1.out‚ 65866.65625! disk.dm-0.out‚225536.24609375$ memory_percent‚92.03426837921143 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.265625 disk.rbd0.in‚0 disk.dm-0.in‚ 8532.81640625! proc.memory.rssMB‚ 2977.140625 disk.sda2.in‚ 133.16796875 disk.md0p1.out‚ 1974.30078125) proc.memory.availableMB‚ 1305.10546875 disk.sdb2.out‚ 2031.08203125! disk.nvme0n1.in‚ 7682.55078125 disk.sdb1.in‚0 disk.rbd2.out‚ 4.150390625$ disk.nvme0n1.out‚159669.58984375% disk./.usageGB‚2721.6755867004395 disk.sda2.out‚ 2031.08203125 disk.md0.out‚ 1974.30078125 disk.sdb.in‚ 75.59765625 cpu‚25.22233315264642 network.recv‚ 4.705792e+06) disk./.usagePercent‚76.09918851387235 disk.sdb2.in‚ 75.59765625 disk.sda.out‚ 2031.08203125ÛUŒ’¨/‚(:‡ ˜ßëÔƒ°×Þ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óâ€s©/‚(:è ˜ßëÔ™²Âà disk.sdb2.in‚ 75.59765625 disk.sda.in‚ 133.16796875! disk.nvme0n1.in‚ 7682.55078125 network.sent‚ 3.1605212e+07 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.out‚2032.583984375 disk.md0p1.in‚ 208.765625 network.recv‚ 4.7102e+06# disk.nvme1n1.out‚65921.69921875 disk.md0p1.out‚ 1975.734375$ memory_percent‚92.03429222106934! proc.memory.rssMB‚ 2977.140625# disk.nvme0n1.out‚159773.3046875( disk./.usagePercent‚76.0843701661447 disk.rbd0.out‚4.375 disk.sda1.out‚0 disk.rbd2.out‚ 4.150390625! disk.dm-0.out‚225695.00390625 disk.sdb.out‚2032.583984375* proc.memory.percent‚18.171024322509766 disk.md0.in‚ 208.765625( proc.memory.availableMB‚ 1305.1015625 cpu‚25.222150990176672 disk.sdb1.in‚0 disk.dm-0.in‚ 8532.8203125 disk.sdb.in‚ 75.59765625$ disk./.usageGB‚2721.145610809326 disk.md0.out‚ 1975.734375 disk.sda.out‚2032.583984375 disk.nvme1n1.in‚ 850.26953125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚ 133.16796875 disk.sdb2.out‚2032.583984375ÆÕ ”ª/‚(:‰ §ßëÔ´˜¾à gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÊà!w«/‚(:ì §ßëÔ´ð“â disk.md0p1.out‚ 1977.3671875 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 7682.55078125 disk.md0.out‚ 1977.3671875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.dm-0.out‚ 225851.046875 disk.sdb.in‚ 75.59765625 cpu‚25.22192779956818! disk.sda.out‚2034.29443359375 disk.nvme1n1.in‚ 850.26953125 disk.rbd0.out‚4.375 disk.sda2.in‚ 133.16796875" disk.sda2.out‚2034.29443359375( proc.memory.availableMB‚ 1305.3515625 proc.cpu.threads‚17 disk.md0p1.in‚ 208.765625* proc.memory.percent‚18.171024322509766 disk.sda1.in‚0" disk.sdb2.out‚2034.29443359375 network.sent‚ 3.1610654e+07 disk.sda.in‚ 133.16796875 disk.nvme1n1.out‚ 65922.15625" disk.nvme0n1.out‚ 159928.890625! disk.sdb.out‚2034.29443359375$ memory_percent‚92.03276634216309 network.recv‚ 4.714006e+06 disk.dm-0.in‚ 8532.8203125! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.08436728631031 disk.rbd2.out‚ 4.150390625 disk.sdb2.in‚ 75.59765625" disk./.usageGB‚2721.1455078125‚wÃâ4¬// _wandbruntime‚14186  _runtime‚14186€mË­/Å ' lr-AdamW/pg1‚1.2906982265399538e-06 ' lr-AdamW/pg2‚1.2906982265399538e-06 trainer/global_step‚11799 % _timestamp‚1.7885388025021644e+09 _runtime‚14186.037754758 _step‚967ÓKÃЮ/Ê % _timestamp‚1.7885388025021644e+09 _runtime‚14186.037754758 _step‚967 ' lr-AdamW/pg1‚1.2906982265399538e-06 ' lr-AdamW/pg2‚1.2906982265399538e-06 trainer/global_step‚11799Çveñg4¯// _wandbruntime‚14186  _runtime‚14186>C_(°/ŠP M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298801317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13515073735168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 298827661 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007409149559808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545475001 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961021051874352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147795704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13514163763712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219278791 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71483089 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007410077534208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1110700423261696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚974484686506249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13514163763712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993895003799040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71483745 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13464542606901 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 246673686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993894985796096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13515073735168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1096588029864448 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1110700678081024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚961021068913712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298827661 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14112648216576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚974485611520613 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13463634631897 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993895003799040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147795702 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚993894985796096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1096588005924352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007410077534208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 246673686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007409149559808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246673684 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 246673684 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219279449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14112417337344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚14186.038652382¾¥÷M(±/P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 545501347 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7885388025029962e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007410077534208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 246673686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007409149559808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246673684 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 246673684 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219279449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14112417337344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚14186.038652382 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298801317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13515073735168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 298827661 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007409149559808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545475001 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961021051874352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147795704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13514163763712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219278791 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71483089 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007410077534208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1110700423261696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚974484686506249 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13514163763712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993895003799040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71483745 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13464542606901 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 246673686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993894985796096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13515073735168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1096588029864448 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1110700678081024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚961021068913712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298827661 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14112648216576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚974485611520613 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13463634631897 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993895003799040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147795702 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚993894985796096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1096588005924352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1110791818466304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545519992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1007491890105856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13464751905175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1096678231628800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961100086508256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219296652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_endÜF×…º/segment.all.freed‚0 trainer/global_step‚11799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993976604881920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993976604881920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147807767 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007492818093568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71489522 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298852518 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1110792071175168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545546364 _step‚969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298826148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13516195208704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚14187.214028346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ØûO;'´/™N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13516195208704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚14187.214028346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147807769 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚974564821374071 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1110791818466304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545519992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1007491890105856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13464751905175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1096678231628800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961100086508256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219296652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993976604881920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993976604881920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147807767 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007492818093568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71489522 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298852518 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1110792071175168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545546364 _step‚969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298826148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615É?µ–24µ// _wandbruntime‚14187  _runtime‚14187¾Wn—\¶/Ö /loss/FRt3/assignment_loss‚1.268713123165071 2loss/SRqqt3/assignment_loss‚0.8387287161895074 0loss/FRt2/assignment_loss‚3.0687525309622288 1loss/SRqqt2/detection_loss‚1.9888161828643398 /loss/SRqqt4/detection_loss‚2.07696523694765 0loss/FBt1/assignment_loss‚0.6764077182635648 .loss/FRt3/detection_loss‚1.543484978217748 0loss/SRqqt1/detection_loss‚2.392156732933862 _step‚970 2loss/SRqqt1/assignment_loss‚1.6880098208785057 /loss/FBt1/detection_loss‚2.0257298913266926 .loss/FBt4/detection_loss‚1.808867993263098 0loss/FRt1/assignment_loss‚3.1909970459730728 1loss/FBt3/assignment_loss‚0.04867197819054108 1loss/SRqqt3/detection_loss‚0.9343460627873852 _runtime‚14187.215561933 1loss/FBt2/assignment_loss‚0.17365335337374496 /loss/FRt1/detection_loss‚3.1516475392424543 /loss/FRt4/detection_loss‚1.9760612216873132 epoch‚9 trainer/global_step‚11799 # _timestamp‚1.78853880368024e+09 /loss/FBt2/detection_loss‚2.4918336922472175 /loss/FBt3/detection_loss‚0.5169020519095162 &loss/total_loss‚40.206222725935746 .loss/FRt2/detection_loss‚2.729688049630803 0loss/FRt4/assignment_loss‚2.5994400588833555 2loss/SRqqt2/assignment_loss‚1.3266447223722935 2loss/SRqqt4/assignment_loss‚1.5093608388588542 1loss/FBt4/assignment_loss‚0.18034318576692515˃Ca·/Û 0loss/FRt1/assignment_loss‚3.1909970459730728 1loss/FBt3/assignment_loss‚0.04867197819054108 1loss/SRqqt3/detection_loss‚0.9343460627873852 _runtime‚14187.215561933 1loss/FBt2/assignment_loss‚0.17365335337374496 /loss/FRt1/detection_loss‚3.1516475392424543 /loss/FRt4/detection_loss‚1.9760612216873132 epoch‚9 trainer/global_step‚11799 # _timestamp‚1.78853880368024e+09 /loss/FBt2/detection_loss‚2.4918336922472175 /loss/FBt3/detection_loss‚0.5169020519095162 &loss/total_loss‚40.206222725935746 .loss/FRt2/detection_loss‚2.729688049630803 0loss/FRt4/assignment_loss‚2.5994400588833555 2loss/SRqqt2/assignment_loss‚1.3266447223722935 2loss/SRqqt4/assignment_loss‚1.5093608388588542 1loss/FBt4/assignment_loss‚0.18034318576692515 /loss/FRt3/assignment_loss‚1.268713123165071 2loss/SRqqt3/assignment_loss‚0.8387287161895074 0loss/FRt2/assignment_loss‚3.0687525309622288 1loss/SRqqt2/detection_loss‚1.9888161828643398 /loss/SRqqt4/detection_loss‚2.07696523694765 0loss/FBt1/assignment_loss‚0.6764077182635648 .loss/FRt3/detection_loss‚1.543484978217748 0loss/SRqqt1/detection_loss‚2.392156732933862 _step‚970 2loss/SRqqt1/assignment_loss‚1.6880098208785057 /loss/FBt1/detection_loss‚2.0257298913266926 .loss/FBt4/detection_loss‚1.808867993263098ÊJa:d”¸/‚(:‰ ¶ßëÔÝ°›à gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³œÆúq¹/‚(:æ ¶ßëÔë¿àâ disk.sdb.in‚ 75.59765625 disk.nvme1n1.in‚ 850.26953125 cpu‚25.221726545353796 disk.rbd2.in‚0 disk.rbd2.out‚ 4.150390625 disk.md0.in‚ 208.765625 disk.sdb.out‚ 2036.3203125$ memory_percent‚92.03593730926514) proc.memory.availableMB‚ 1304.83203125! proc.memory.rssMB‚ 2977.140625 network.sent‚ 3.1626959e+07 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.out‚ 2036.3203125 disk.sda.out‚ 2036.3203125 disk.md0p1.out‚ 1979.32421875" disk.nvme0n1.out‚ 160032.546875 disk.dm-0.in‚ 8532.8203125# disk.nvme1n1.out‚66009.09765625 disk.rbd0.in‚0 disk.rbd0.out‚4.375 network.recv‚ 4.718083e+06# disk./.usageGB‚2721.17293548584 disk.sdb1.in‚0! disk.dm-0.out‚226041.64453125 disk.sda.in‚ 133.16796875) disk./.usagePercent‚76.08513417553627! disk.nvme0n1.in‚ 7682.55078125 disk.md0.out‚ 1979.32421875* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.sda2.out‚ 2036.3203125 disk.sdb2.in‚ 75.59765625 disk.md0p1.in‚ 208.765625 disk.sda1.out‚0 disk.sda2.in‚ 133.1679687571’’º/‚(:‡ ÅßëÔüãÞà gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ·²uw»/‚(:ì ÅßëÔ¯Ú„á disk.sda2.out‚2037.712890625 network.sent‚ 3.1724354e+07 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda.out‚2037.712890625 disk.nvme1n1.in‚ 850.26953125( proc.memory.availableMB‚ 1305.3203125! proc.memory.rssMB‚ 2977.140625) disk./.usagePercent‚76.08520915789036 disk.sdb2.in‚ 75.59765625 disk.sdb2.out‚2037.712890625 disk.sda.in‚ 133.16796875 disk.rbd0.out‚4.375 cpu‚25.22151682982544 disk.rbd2.out‚ 4.150390625 disk.sdb.in‚ 75.59765625 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚92.03295707702637 proc.cpu.threads‚17% disk./.usageGB‚2721.1756172180176 disk.sda1.out‚0 disk.md0p1.out‚ 1980.6484375* proc.memory.percent‚18.171024322509766# disk.nvme1n1.out‚66009.33203125 network.recv‚ 4.725502e+06! disk.nvme0n1.in‚ 7682.55078125 disk.sda2.in‚ 133.16796875 disk.md0.in‚ 208.765625! disk.dm-0.out‚226145.66015625 disk.md0p1.in‚ 208.765625" disk.nvme0n1.out‚ 160136.328125 disk.md0.out‚ 1980.6484375 disk.dm-0.in‚ 8532.8203125 disk.sdb.out‚2037.712890625 disk.rbd0.in‚0Â+² ”¼/‚(:‰ ÔßëÔˆÞ¦à gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°®>y½/‚(:î ÔßëÔè‡àá network.sent‚ 3.1729592e+07 disk.md0p1.out‚ 1982.1484375 disk.sda1.in‚0* proc.memory.percent‚18.171024322509766 disk.sda1.out‚0) disk./.usagePercent‚76.08870335692266% disk./.usageGB‚2721.3005867004395! disk.dm-0.out‚226317.26171875 disk.sdb.in‚ 75.59765625 disk.rbd2.out‚ 4.150390625! proc.memory.rssMB‚ 2977.140625! disk.nvme0n1.in‚ 7682.55078125" disk.nvme0n1.out‚ 160239.984375 disk.dm-0.in‚ 8532.8203125$ memory_percent‚92.03453063964844 cpu‚25.221333523045544 proc.cpu.threads‚17 disk.sdb1.out‚0! disk.sdb2.out‚2039.2763671875 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.26953125# disk.nvme1n1.out‚66077.27734375 disk.md0p1.in‚ 208.765625 disk.sdb1.in‚0% proc.memory.availableMB‚ 1305.0625 network.recv‚ 4.729245e+06 disk.sda2.in‚ 133.16796875! disk.sda2.out‚2039.2763671875 disk.md0.out‚ 1982.1484375 disk.sdb.out‚2039.2763671875 disk.sda.out‚2039.2763671875 disk.rbd2.in‚0 disk.md0.in‚ 208.765625 disk.sdb2.in‚ 75.59765625 disk.rbd0.in‚0 disk.rbd0.out‚4.375Dr™’¾/‚(:‡ ãßëÔ¨Çß gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qp‰|¿/‚(:ñ ãßëÔ¬ù€á disk.md0p1.out‚ 1983.5703125$ disk.nvme0n1.out‚160447.35546875 cpu‚25.221138459877313$ disk./.usageGB‚2721.425849914551 disk.rbd2.out‚ 4.150390625! disk.nvme0n1.in‚ 7682.55078125! disk.dm-0.out‚226557.58984375* proc.memory.percent‚18.171024322509766 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.md0.out‚ 1983.5703125 disk.dm-0.in‚ 8532.8203125 disk.nvme1n1.in‚ 850.26953125 network.recv‚ 4.733232e+06 disk.md0p1.in‚ 208.765625 disk.sda1.in‚0 disk.sdb.out‚2040.7470703125 disk.sda.in‚ 133.16796875$ memory_percent‚92.03145503997803! proc.memory.rssMB‚ 2977.140625 disk.sda1.out‚0 disk.rbd2.in‚0! disk.nvme1n1.out‚ 66110.234375! disk.sdb2.out‚2040.7470703125) disk./.usagePercent‚76.09220576881594 disk.rbd0.in‚0! disk.sda2.out‚2040.7470703125 disk.md0.in‚ 208.765625 disk.rbd0.out‚4.375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚2040.7470703125) proc.memory.availableMB‚ 1305.56640625 network.sent‚ 3.1734994e+07 disk.sdb.in‚ 75.59765625 disk.sdb2.in‚ 75.59765625ãOTÚ4À// _wandbruntime‚14246  _runtime‚14246àÃÅÁ/¿ % lr-AdamW/pg1‚8.85427647659745e-07 % lr-AdamW/pg2‚8.85427647659745e-07 trainer/global_step‚11849 # _timestamp‚1.78853886270073e+09 _runtime‚14246.236199062 _step‚971†Íi˜ÊÂ/Ä # _timestamp‚1.78853886270073e+09 _runtime‚14246.236199062 _step‚971 % lr-AdamW/pg1‚8.85427647659745e-07 % lr-AdamW/pg2‚8.85427647659745e-07 trainer/global_step‚11849Ë]f_4Ã// _wandbruntime‚14246  _runtime‚14246ë¸Â"˜Ä/‰P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300046219 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520419862009 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1011547119250432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13570251188736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300072563 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 148398952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚14246.24017195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13571161160192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚997975958090240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1011546191276032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14171111218176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚964971948640912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 247681686 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519511887005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚997975940087296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220172977 ]ODeviceStatsMonitor.on_train_batch_start/i¥wnactive_split_bytes.small_pool.current‚ 235073536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148398954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71774025 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885388627014575e+09 _step‚972 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚997975958090240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547754249 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1011546191276032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964971931601552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220173605 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1101099315086848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚11849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115270402364928 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1011547119250432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 247681684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300046219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14171346291712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 300072563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247681686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71774651f9ŒJ(Å/ŽP \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚997975940087296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13571161160192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 300072563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247681686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71774651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300046219 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520419862009 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1011547119250432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13570251188736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300072563 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 148398952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚14246.24017195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13571161160192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚997975958090240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1011546191276032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14171111218176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚964971948640912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 247681686 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519511887005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚997975940087296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220172977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148398954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71774025 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885388627014575e+09 _step‚972 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚997975958090240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547754249 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1011546191276032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964971931601552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220173605 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1101099315086848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚11849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115270402364928 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1011547119250432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 247681684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300046219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14171346291712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547799266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚998057577176064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1011628931822080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247701844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 547772894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247701846 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14172297115648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1115361813966848 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101189540791296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13571372648960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14172515398656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547799266 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998057577176064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965050966235456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300097420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300097420 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1115362056189952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚998057559173120 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148411019 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13572282633728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 247701844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 220191381 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978572503370951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300071050 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220190759 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13520629160283 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885388637005675e+09 _step‚973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚14247.237598531 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 trainer/global_step‚11849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300071050 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13571372648960 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 247701846 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 547772894 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1101189516851200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978571578356379é„öÝÈ/™N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547799266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚998057577176064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1011628931822080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247701844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 547772894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247701846 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14172297115648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1115361813966848 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101189540791296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13571372648960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14172515398656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547799266 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998057577176064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965050966235456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300097420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300097420 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1115362056189952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚998057559173120 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148411019 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13572282633728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 247701844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 220191381 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978572503370951 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.frÞh´W eed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300071050 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220190759 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13520629160283 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885388637005675e+09 _step‚973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _runtime‚14247.237598531 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 trainer/global_step‚11849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300071050 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13571372648960 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 247701846 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 547772894 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1101189516851200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978571578356379 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015683232992256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221066827 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1119840360201728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 301317465 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13626338613760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 301291121 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 149002204 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944xƒ!(Ù/“P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982498200470865 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13627248585216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002056912381440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13627248585216 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1105610576369152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 248689684 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13626338613760 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1105610600309248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚968922828368112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002056894378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015683232992256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221066827 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1119840360201728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 301317465 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13626338613760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 301291121 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 149002204 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885389228106813e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13576297117117 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1119840604535296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72069744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 301342322 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002138531467264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13628370058752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1119932004349952 trainer/global_step‚11899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301315952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221084011 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14231178336256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚969001828923296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13576506415391 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚977 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550052168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002138513464320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13627460073984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119931743252480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14230941178880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 550025796 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1105700802073600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 550052168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550025796 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301315952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221084639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002138513464320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982579260353259 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015765973538304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚969001845962656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 149014267 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760,üø„#'Ü/N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550052168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002138513464320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13627460073984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119931743252480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14230941178880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 550025796 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1105700802073600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 550052168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550025796 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301315952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221084639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002138513464320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982579260353259 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015765973538304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚969001845962656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 149014267 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 248709844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 301342322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 248709846 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 248709844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885389240170686e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13627460073984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 248709846 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72069744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 301342322 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002138531467264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13628370058752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1119932004349952 trainer/global_step‚11899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301315952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221084011 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14231178336256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚969001828923296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13576506415391 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚977 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221959993 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 552233707 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 302536023 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚972873708095312 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1019821202682880 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1019820274708480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13631266397221 _runtime‚14360.527991746 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1006137866672640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13632174372225 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885389769922404e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 302536023 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1019820274708480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72354541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972873691055952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13683336010240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 249697684 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚986505882467537 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1124410570399232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221960603 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1019820274708480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13631266397221 _runtime‚14360.527991746 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1006137866672640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13632174372225 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885389769922404e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 302536023 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1019820274708480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72354541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972873691055952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13683336010240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 249697684 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚986505882467537 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1124410570399232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221960603 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221959993 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 552233707 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 302536023 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚972873708095312Ô{Ô¿¶4î// _wandbruntime‚14361  _runtime‚14361DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221977890 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 552278698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72360373 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 552278698 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 _runtime‚14361.499102236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1124502209675264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13684457483776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302560854 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13683547499008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885389779628851e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 302587224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚981 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1006219467755520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13683547499008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72360988 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚986586017335567 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552305070 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712ÓRŽ/Yð/N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302587224 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221978507 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 249717846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14290098439168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1019903943242240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1019903015254528 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 149617519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13683547499008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885389779628851e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 302587224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚981 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1006219467755520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13683547499008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72360988 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚986586017335567 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552305070 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302560854 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 552305070 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 249717844 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 149617517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972952725689856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚”ùQXÊ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13684457483776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 249717844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13632383670499 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221977890 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 552278698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72360373 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 552278698 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 368193024 _runtime‚14361.499102236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1124502209675264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13684457483776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302560854 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1023957316424704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13739423435264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554486609 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚990511714435481 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788539029960105e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 250705684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 250705684 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1128980511253504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1114633146813952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚976824570783152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303780925 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1114633170754048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554512955 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 _runtime‚14413.495937613 _step‚984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14347364439552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13738513463808 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1023958244399104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72645944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303780925 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1010218802960896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150208704 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150208702 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 trainer/global_step‚11999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1023957316424704 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303807269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010218820963840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72645329 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13739423435264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1010218802960896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚990512639449845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1023958244399104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010218820963840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096@†¡%ÿ/’P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1023958244399104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010218820963840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13738513463808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222854648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1023957316424704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13739423435264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554486609 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚990511714435481 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788539029960105e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 250705684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 250705684 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1128980511253504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1114633146813952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚976824570783152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303780925 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1114633170754048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554512955 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 _runtime‚14413.495937613 _step‚984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14347364439552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13738513463808 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1023958244399104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72645944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303780925 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1010218802960896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150208704 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150208702 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 trainer/global_step‚11999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1023957316424704 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303807269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010218820963840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72645329 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.larvÈówge_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13739423435264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1010218802960896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚990512639449845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615Ø ¦4€0/ _wandbruntime‚14414  _runtime‚14414ï=„;'0—N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303832126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303805756 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13740544908800 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1114723396458496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554557972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 250725846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222872025 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129072160808448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚976903588377696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554557972 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚990592774317875 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72651879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885390311219501e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13740544908800 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 250725846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150220769 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150220767 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13688260925607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615m v"'‚0œN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885390311219501e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13740544908800 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 250725846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150220769 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150220767 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13688260925607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303832126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303805756 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13740544908800 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1114723396458496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554557972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 250725846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222872025 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129072160808448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚976903588377696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554557972 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚990592774317875 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72651879 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185Ùá2#4ƒ0/ _wandbruntime‚14414  _runtime‚14414A¸ÒNb„0Ü 1loss/FBt2/assignment_loss‚0.18642947117950825 1loss/FBt4/assignment_loss‚0.27471226162646145 /loss/FRt3/detection_loss‚1.1155156070739032 trainer/global_step‚11999 2loss/FBt3/assignment_loss‚0.034545745876792294 .loss/FRt1/detection_loss‚3.101272299333855 1loss/SRqqt2/detection_loss‚2.0750056622843993 1loss/SRqqt4/detection_loss‚1.9849631413817406 0loss/FRt4/assignment_loss‚3.2410178853541005 .loss/FRt2/detection_loss‚2.598812621512237 /loss/FBt1/detection_loss‚1.9779824632055618 /loss/FBt3/detection_loss‚0.5490793153467411 &loss/total_loss‚38.762302236264986 _runtime‚14414.659583835 2loss/SRqqt3/assignment_loss‚0.9733806607934335 % _timestamp‚1.7885390311241956e+09 0loss/FRt3/assignment_loss‚1.1749804957707723 1loss/SRqqt3/detection_loss‚0.6847204893711023 .loss/FBt4/detection_loss‚1.935950055718422 epoch‚9 1loss/SRqqt2/assignment_loss‚1.084058465485118 /loss/FRt4/detection_loss‚2.0640897042289073 1loss/SRqqt1/detection_loss‚2.3205771416425707 /loss/FBt2/detection_loss‚1.6582576259970665 0loss/FRt1/assignment_loss‚2.9330875939793057 2loss/SRqqt1/assignment_loss‚1.5126866887013117 1loss/FBt1/assignment_loss‚0.23590907352200596 _step‚986 0loss/FRt2/assignment_loss‚3.4047291696071627 1loss/SRqqt4/assignment_loss‚1.640538597272502,ÁXÕg…0á 2loss/FBt3/assignment_loss‚0.034545745876792294 .loss/FRt1/detection_loss‚3.101272299333855 1loss/SRqqt2/detection_loss‚2.0750056622843993 1loss/SRqqt4/detection_loss‚1.9849631413817406 0loss/FRt4/assignment_loss‚3.2410178853541005 .loss/FRt2/detection_loss‚2.598812621512237 /loss/FBt1/detection_loss‚1.9779824632055618 /loss/FBt3/detection_loss‚0.5490793153467411 &loss/total_loss‚38.762302236264986 _runtime‚14414.659583835 2loss/SRqqt3/assignment_loss‚0.9733806607934335 % _timestamp‚1.7885390311241956e+09 0loss/FRt3/assignment_loss‚1.1749804957707723 1loss/SRqqt3/detection_loss‚0.6847204893711023 .loss/FBt4/detection_loss‚1.935950055718422 epoch‚9 1loss/SRqqt2/assignment_loss‚1.084058465485118 /loss/FRt4/detection_loss‚2.0640897042289073 1loss/SRqqt1/detection_loss‚2.3205771416425707 /loss/FBt2/detection_loss‚1.6582576259970665 0loss/FRt1/assignment_loss‚2.9330875939793057 2loss/SRqqt1/assignment_loss‚1.5126866887013117 1loss/FBt1/assignment_loss‚0.23590907352200596 _step‚986 0loss/FRt2/assignment_loss‚3.4047291696071627 1loss/SRqqt4/assignment_loss‚1.640538597272502 1loss/FBt2/assignment_loss‚0.18642947117950825 1loss/FBt4/assignment_loss‚0.27471226162646145 /loss/FRt3/detection_loss‚1.1155156070739032 trainer/global_step‚11999Ú ¼f’†0‚(:‡ —áëÔäÕÞß gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hkg±u‡0‚(:ê —áëÔØÅžà disk.rbd2.in‚0 disk.md0p1.out‚ 2004.921875 disk.rbd0.out‚ 4.44921875* proc.memory.percent‚18.171024322509766 disk.rbd2.out‚ 4.162109375 disk.sdb2.in‚ 75.59765625 disk.sda.out‚2062.837890625 disk.nvme1n1.in‚ 850.48828125 disk.sda1.out‚0$ disk.nvme0n1.out‚161706.04296875 disk.dm-0.in‚ 8534.44140625! disk.dm-0.out‚228495.13671875! proc.memory.rssMB‚ 2977.140625 disk.sda1.in‚0 disk.nvme0n1.in‚ 7683.953125 disk.md0.out‚ 2004.921875 disk.sdb.out‚2062.837890625 disk.md0p1.in‚ 208.765625% disk./.usageGB‚2720.1703567504883 disk.md0.in‚ 208.765625 disk.nvme1n1.out‚ 66789.09375 proc.cpu.threads‚17 disk.sda2.in‚ 133.16796875 disk.sdb.in‚ 75.59765625 disk.rbd0.in‚0% proc.memory.availableMB‚ 1304.6875 disk.sda2.out‚2062.837890625) disk./.usagePercent‚76.05710165448404 disk.sdb1.in‚0 disk.sdb2.out‚2062.837890625 cpu‚25.21860959387177 network.sent‚ 3.2112552e+07 disk.sda.in‚ 133.16796875$ memory_percent‚92.03681945800781 network.recv‚ 4.798656e+06 disk.sdb1.out‚0—[óT”ˆ0‚(:‰ ¦áëÔ­¥–à gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0**P݉0‚(:ö ¦áëÔÛéåà disk.sda2.in‚ 133.16796875 disk.md0.out‚ 2006.72265625 cpu‚25.218415250672543 proc.cpu.threads‚17 disk.rbd0.out‚ 4.44921875! disk.sda2.out‚2064.6962890625 disk.sda1.in‚0 disk.nvme0n1.in‚ 7683.953125 disk.sdb.out‚2064.6962890625 disk.sda.out‚2064.6962890625 network.recv‚ 4.80532e+06) disk./.usagePercent‚76.06071275345955! disk.sdb2.out‚2064.6962890625$ memory_percent‚92.03557968139648 disk.sda1.out‚0 disk.rbd2.in‚0% disk./.usageGB‚2720.2995071411133 disk.md0p1.out‚ 2006.72265625$ disk.nvme0n1.out‚161782.96484375! proc.memory.rssMB‚ 2977.140625 disk.rbd0.in‚0! disk.dm-0.out‚228572.23828125 disk.sdb.in‚ 75.59765625 disk.sda.in‚ 133.16796875 disk.nvme1n1.in‚ 850.48828125" disk.nvme1n1.out‚ 66789.2734375 disk.md0p1.in‚ 208.765625 network.sent‚ 3.2222313e+07 disk.sdb2.in‚ 75.59765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 208.765625' proc.memory.availableMB‚ 1304.890625* proc.memory.percent‚18.171024322509766 disk.dm-0.in‚ 8534.44140625 disk.rbd2.out‚ 4.224609375Àí=g“Š0‚(:ˆ µáëÔП£à gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 iˆþ}‹0‚(:ò µáëÔ¡«¼á disk.dm-0.in‚ 8534.44140625 disk.md0p1.in‚ 208.765625 disk.rbd0.out‚ 4.55126953125! proc.memory.rssMB‚ 2977.140625 disk.dm-0.out‚ 228747.0625! disk.sdb2.out‚2066.2080078125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1304.63671875 disk.rbd0.in‚0 disk.rbd2.out‚ 4.224609375 disk.nvme0n1.in‚ 7683.953125$ memory_percent‚92.03712940216064 proc.cpu.threads‚17 disk.sdb1.out‚0* proc.memory.percent‚18.171024322509766 cpu‚25.218203687293617 disk.sda1.out‚0 disk.sda2.in‚ 133.16796875 network.recv‚ 4.809804e+06) disk./.usagePercent‚76.06595362536589$ disk./.usageGB‚2720.486946105957 disk.sdb2.in‚ 75.59765625 disk.sda.out‚2066.2080078125 disk.sda.in‚ 133.16796875 disk.md0p1.out‚ 2008.171875 disk.sdb1.in‚0 disk.sdb.in‚ 75.59765625" disk.nvme1n1.out‚ 66789.5234375 disk.sda1.in‚0# disk.nvme0n1.out‚161957.5390625 disk.md0.out‚ 2008.171875 network.sent‚ 3.2226317e+07 disk.sdb.out‚2066.2080078125 disk.nvme1n1.in‚ 850.48828125! disk.sda2.out‚2066.2080078125 disk.md0.in‚ 208.765625ûODžŒ0‚(:“ ÄáëÔ´š–à gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ꮩ‡0‚(:ü ÄáëԉУà network.recv‚ 4.813651e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 75.59765625# disk.nvme1n1.out‚66789.73828125 cpu‚25.217982073832594 proc.cpu.threads‚17) disk./.usagePercent‚76.07822758609979 disk.nvme0n1.in‚ 7683.953125! disk.dm-0.out‚228839.19921875 disk.md0p1.in‚ 208.765625 disk.nvme1n1.in‚ 850.48828125 disk.md0p1.out‚ 2009.59765625* proc.memory.percent‚18.171024322509766 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 2009.59765625 disk.dm-0.in‚ 8534.44140625! disk.sdb2.out‚2067.6884765625! disk.sda2.out‚2067.6884765625 disk.sdb.in‚ 75.59765625$ memory_percent‚92.03860759735107 disk.rbd0.out‚ 4.55126953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 133.16796875! proc.memory.rssMB‚ 2977.140625 disk.rbd0.in‚0# disk.nvme0n1.out‚162049.4609375 disk.md0.in‚ 208.765625 disk.sda.in‚ 133.16796875 disk.sda.out‚2067.6884765625 network.sent‚ 3.2232385e+07$ disk./.usageGB‚2720.925922393799 disk.rbd2.out‚ 4.3037109375 disk.sdb.out‚2067.6884765625) proc.memory.availableMB‚ 1304.39453125¿×ØX4Ž0/ _wandbruntime‚14467  _runtime‚14467ÀÌþ/Ä0¾ $ lr-AdamW/pg1‚2.5636389952888e-08 $ lr-AdamW/pg2‚2.5636389952888e-08 trainer/global_step‚12049 $ _timestamp‚1.788539084403037e+09 _runtime‚14467.938684462 _step‚987»»ÓÂÉ0à $ lr-AdamW/pg1‚2.5636389952888e-08 $ lr-AdamW/pg2‚2.5636389952888e-08 trainer/global_step‚12049 $ _timestamp‚1.788539084403037e+09 _runtime‚14467.938684462 _step‚987Û£…E±4‘0/ _wandbruntime‚14467  _runtime‚14467§½V0G ’0P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13794600888832 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚994519396432153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 251713686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 305052171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 251713684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72938677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13743020907437 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1133550999603712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes¸šHPÌ.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 305025827 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1133550746881536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14406543627264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚994518471417789 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚980775467549712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 251713684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚988 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150811954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 305052171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13743928882441 $ _timestamp‚1.788539084403987e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 305025827 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223750631 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72938031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1014299775255040 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1014299757252096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223749983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚980775450510352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 556765857 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13795510860288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13795510860288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1119144432036352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14406314845184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚14467.939730716 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 556739511 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1028094358140928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚980775450510352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 556765857 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13795510860288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13795510860288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1119144432036352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14406314845184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚14467.939730716 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 556739511 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1028094358140928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 251713686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 305052171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 251713684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72938677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13743020907437 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1133550999603712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 305025827 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1133550746881536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14406543627264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚994518471417789 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚980775467549712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 251713684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚988 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150811954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 305052171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13743928882441 $ _timestamp‚1.788539084403987e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 305025827 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223750631 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72938031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784ÜI•Ñ4”0/ _wandbruntime‚14469  _runtime‚14469ˆRšß'•0˜N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 251733844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1014381394340864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1028178026674688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚12049 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150824019 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 305077028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14407716662272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 251733846 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚14469.008178728 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223767855 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 251733844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72944476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1014381376337920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556810874 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72943838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚980854468104896 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556784502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚980854485144256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚994599531300183 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305050658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1119234681680896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13745046155927 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13796632333824 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1119234657740800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13744138180715 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 251733846 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚14469.008178728 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223767855 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 251733844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72944476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1014381376337920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556810874 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72943838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚980854468104896 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556784502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚980854485144256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚994599531300183 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305050658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_2u(j|end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1119234681680896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13745046155927 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13796632333824 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1119234657740800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13744138180715 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1028178026674688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚12049 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150824019 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 305077028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14407716662272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13813683857063 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 306482761 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1020304924214272 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14478605489152 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 559802042 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 225211312 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1020304942217216 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1034169758141440 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1140079401543680 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚986486068080032 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 368193024 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1125600796054528 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚237 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚73436157 trainer/global_step‚12089 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1034170459183616 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14478790670848 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1020304942217216 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13865516966400 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1034170459183616 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 306482761 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 209121792 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 559781329 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 [FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚1000299053830287 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚13813002789615 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232ß'P_¹4«0/ _wandbruntime‚14575  _runtime‚14575Ô’î†)¬0€S SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 253314046 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 306491575 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 209121792 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 185181696 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13813491000499 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 YDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚1000349968146355 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1020359185121280 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 306512286 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚986536477145856 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788539191513221e+09 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14479280521728 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1020359185121280 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 151784717 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 253314046 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 306491575 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14479095340032 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1020359203124224 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 559805619 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 trainer/global_step‚12089 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1140136158948352 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚986536494185216 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13866006817280 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚73441861 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559826332 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 253314044 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚73441624 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 _runtime‚14575.049440165 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1125657087548416 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225226580 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1000350666253163 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 253314044 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1034225209941504 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1034224508899328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13865323778048 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1140136368070144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185FKXþ‹)­0…S JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788539191513221e+09 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14479280521728 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1020359185121280 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 151784717 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 253314046 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 306491575 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14479095340032 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1020359203124224 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 559805619 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 trainer/global_step‚12089 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1140136158948352 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚986536494185216 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13866006817280 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚73441861 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559826332 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 253314044 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚73441624 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 _runtime‚14575.049440165 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1125657087548416 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225226580 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1000350666253163 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 253314044 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1034225209941504 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1034224508899328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13865323778048 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1140136368070144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 253314046 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 306491575 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 209121792 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 185181696 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13813491000499 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 YDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚1000349968146355 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1020359185121280 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 306512286 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚986536477145856 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0à{|ÅÆ“®0‚(:ˆ ¼âëÔô•á gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ra‡ v¯0‚(:ë ¼âëÔ¤«Þá! disk.nvme1n1.out‚ 66791.796875 disk.md0p1.out‚ 2022.3984375 disk.sda2.in‚ 133.16796875# memory_percent‚92.0375108718872 proc.cpu.threads‚17 disk.rbd0.out‚ 4.716796875 disk.sda1.out‚0 disk.md0.in‚ 208.765625 disk.sda2.out‚ 2080.94921875 disk.sdb.out‚ 2080.94921875 disk.sdb2.in‚ 75.59765625 disk.sdb2.out‚ 2080.94921875 disk.sda.out‚ 2080.94921875 disk.nvme1n1.in‚ 850.48828125 disk.sdb1.out‚0 disk.rbd2.out‚ 4.4296875( disk./.usagePercent‚76.0851023906977 disk.sda.in‚ 133.16796875 disk.md0.out‚ 2022.3984375) proc.memory.availableMB‚ 1304.57421875! disk.dm-0.out‚230509.12890625$ disk.nvme0n1.out‚163717.33203125 cpu‚25.234896358504773 network.sent‚ 3.2397225e+07% disk./.usageGB‚2721.1717987060547 disk.sdb.in‚ 75.59765625 disk.sda1.in‚0 network.recv‚ 4.856074e+06 disk.sdb1.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2977.140625 disk.dm-0.in‚ 8534.44140625 disk.md0p1.in‚ 208.765625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 7683.953125* proc.memory.percent‚18.171024322509766YQ~Ž4°0/ _wandbruntime‚14591  _runtime‚14591aÞ`˜­±0Ǧ ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5048211216926575 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38487252593040466 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.027253268286585808 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19215898215770721 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1781904250383377 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.5055555701255798 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9167219400405884 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920836091041565 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.507927417755127 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916111826896667 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/0FRtÙCÓ|ù3SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24074804782867432 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.556839108467102 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 trainer/global_step‚12089 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5076271295547485 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 *jet/accuracy_1_of_2‚0.7458094954490662 particle/accuracy_3_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17016929388046265 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.479051798582077 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987620711326599 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916492700576782 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6451472043991089 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5740740895271301 +jet/accuracy_2_of_4‚0.41270336508750916 particle/accuracy_3_of_5‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5738567113876343 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5041050910949707 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4829712510108948 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33811330795288086 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚14591.210856228 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2659970819950104 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48646607995033264 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1515074521303177 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15580809116363525 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15310394763946533 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18072961270809174 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41111108660697937 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.450762540102005 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24578440189361572 +particle/specificity‚0.9480957984924316 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9937856197357178 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18474997580051422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883157014846802 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961804986000061 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,particle/sensitivity‚0.33990731835365295 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38553309440612793 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.21929825842380524 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4767831861972809 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.555440366268158 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1580459624528885 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 epoch‚9 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48424670100212097 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5062536597251892 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.399219810962677 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10853956639766693 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4793072044849396 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.277935653924942 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.44675928354263306 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03231268748641014 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5545977354049683 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 % _timestamp‚1.7885392076726356e+09 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.464074045419693 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2900221645832062 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5830100178718567 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5241379141807556 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12274409085512161 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18893364071846008 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4729127287864685 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_3_of_4‚0.08796161413192749 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.512069821357727 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1648697406053543 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5753994584083557 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24998705089092255 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9756944179534912 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28661012649536133 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14541131258010864 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08408021181821823 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4774716794490814 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37514880299568176 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11762946099042892 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4226934313774109 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014942528679966927 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 jet/accuracy_5_of_6‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29114365577697754 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24294346570968628 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10997506231069565 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_5_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 jet/accuracy_6_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988539814949036 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2604166865348816 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42869386076927185 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19643792510032654 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4682241976261139 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38963818550109863 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10116958618164062 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.54666668176651 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5747338533401489 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906195998191833 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09026434272527695 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.034688349813222885 6validation_average_jet_accuracy‚0.4336489737033844 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358506679535 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841206073760986 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4801693856716156 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38860467076301575 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4766082167625427 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5002261996269226 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4922945201396942 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02418098971247673 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2434116005897522 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5311294198036194 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20540495216846466 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1537131667137146 (particle/accuracy‚0.8531384468078613 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19422827661037445 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07619182020425797 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1706349104642868 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4636562168598175 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.66865074634552 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5626010298728943 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909769296646118 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.4249999225139618 particle/accuracy_2_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49815070629119873 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2080114781856537 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8208389282226562 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1442427784204483 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882344007492065 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02596586011350155 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5189814567565918 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08385305106639862 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18072961270809174 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23551909625530243 *jet/accuracy_1_of_3‚0.8660643100738525 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5075494647026062 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023963134735822678 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5503039956092834 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5073954463005066 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6422618627548218 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5253929495811462 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5196970105171204 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.249672070145607 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49817705154418945 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4595407247543335 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.46870675683021545 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6503087282180786 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2177303433418274 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06964285671710968 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.946601390838623 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 particle/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1756504625082016 + Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5625 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_2‚1 particle/accuracy_6_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04802677035331726 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010642197914421558 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02721637859940529 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4848484992980957 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17745612561702728 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12020960450172424 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03366174176335335 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9902740716934204 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6075867414474487 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01787603832781315 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1862524002790451 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9908159971237183 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4858555197715759 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27781441807746887 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47490403056144714 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11867816746234894 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1773967146873474 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5605034828186035 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5754227638244629 particle/accuracy_1_of_1‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4697107970714569 +jet/accuracy_3_of_5‚0.09599973261356354 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5546659231185913 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14965508878231049 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913790822029114 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 (loss/val_total_loss‚39.9488425724643 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907159805297852 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17259719967842102 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.713651716709137 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.375343918800354 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_4‚0.00578314159065485 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5083449482917786 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.440849632024765 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3058599531650543 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47892725467681885 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2569444179534912 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 particle/accuracy_1_of_3‚1 _step‚993 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.596977949142456 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4707582890987396 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018611295148730278 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18230733275413513 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_2_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2646864950656891 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21617849171161652 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 +jet/accuracy_4_of_6‚0.01923076994717121 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004440665245056152 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15329217910766602 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590277314186096 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870032072067261 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893686175346375 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"PuritâŸf¨ y/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3203703761100769 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3349684476852417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9763889312744141 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5271317958831787 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 (particle/f_score‚0.41936561465263367 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00611328799277544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5515779256820679 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15247748792171478 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22911739349365234 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4939053952693939 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.016308244317770004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027418745681643486 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18000002205371857 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017636684700846672 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4640471339225769 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3483099639415741 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4636562168598175 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44642862677574160Õ!­Jr²0̦ =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.713651716709137 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.375343918800354 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_4‚0.00578314159065485 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5083449482917786 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.440849632024765 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3058599531650543 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47892725467681885 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2569444179534912 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 particle/accuracy_1_of_3‚1 _step‚993 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.596977949142456 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4707582890987396 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018611295148730278 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18230733275413513 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_2_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2646864950656891 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21617849171161652 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 +jet/accuracy_4_of_6‚0.01923076994717121 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004440665245056152 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15329217910766602 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590277314186096 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870032072067261 particle/accuracy_3_of_3‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893686175346375 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3203703761100769 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3349684476852417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9763889312744141 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5271317958831787 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 (particle/f_score‚0.41936561465263367 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00611328799277544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5515779256820679 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15247748792171478 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22911739349365234 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4939053952693939 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.016308244317770004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027418745681643486 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18000002205371857 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017636684700846672 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4640471339225769 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3483099639415741 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4636562168598175 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4464286267757416 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5048211216926575 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38487252593040466 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.027253268286585808 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19215898215770721 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1781904250383377 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.5055555701255798 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9167219400405884 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920836091041565 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.507927417755127 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916111826896667 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24074804782867432 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.556839108467102 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 trainer/global_step‚12089 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5076271295547485 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 *jet/accuracy_1_of_2‚0.7458094954490662 particle/accuracy_3_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17016929388046265 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.479051798582077 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987620711326599 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916492700576782 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6451472043991089 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5740740895271301 +jet/accuracy_2_of_4‚0.41270336508750916 particle/accuracy_3_of_5‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5738567113876343 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5041050910949707 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4829712510108948 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33811330795288086 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚14591.210856228 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2659970819950104 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48646607995033264 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1515074521303177 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15580809116363525 jet/accuracy_6_of_6‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15310394763946533 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18072961270809174 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41111108660697937 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.450762540102005 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24578440189361572 +particle/specificity‚0.9480957984924316 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9937856197357178 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18474997580051422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883157014846802 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5961804986000061 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,particle/sensitivity‚0.33990731835365295 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38553309440612793 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.21929825842380524 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4767831861972809 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.555440366268158 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1580459624528885 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 epoch‚9 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48424670100212097 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5062536597251892 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.399219810962677 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10853956639766693 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4793072044849396 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.277935653924942 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.44675928354263306 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03231268748641014 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5545977354049683 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 % _timestamp‚1.7885392076726356e+09 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.464074045419693 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2900221645832062 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5830100178718567 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5241379141807556 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12274409085512161 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18893364071846008 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4729127287864685 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_3_of_4‚0.08796161413192749 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.512069821357727 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1648697406053543 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9885057210922241 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5753994584083557 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24998705089092255 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9756944179534912 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28661012649536133 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14541131258010864 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08408021181821823 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4774716794490814 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37514880299568176 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11762946099042892 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4226934313774109 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014942528679966927 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 jet/accuracy_5_of_6‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29114365577697754 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24294346570968628 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10997506231069565 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_5_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 jet/accuracy_6_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988539814949036 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2604166865348816 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42869386076927185 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19643792510032654 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4682241976261139 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38963818550109863 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10116958618164062 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.54666668176651 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5747338533401489 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906195998191833 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09026434272527695 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.034688349813222885 6validation_average_jet_accuracy‚0.4336489737033844 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358506679535 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841206073760986 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4801693856716156 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38860467076301575 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4766082167625427 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5002261996269226 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4922945201396942 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02418098971247673 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2434116005897522 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5311294198036194 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20540495216846466 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1537131667137146 (particle/accuracy‚0.8531384468078613 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19422827661037445 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07619182020425797 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1706349104642868 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4636562168598175 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.66865074634552 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5626010298728943 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909769296646118 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.4249999225139618 particle/accuracy_2_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49815070629119873 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2080114781856537 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8208389282226562 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1442427784204483 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Puáá2 !rity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882344007492065 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02596586011350155 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5189814567565918 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08385305106639862 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18072961270809174 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23551909625530243 *jet/accuracy_1_of_3‚0.8660643100738525 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5075494647026062 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023963134735822678 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5503039956092834 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5073954463005066 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6422618627548218 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5253929495811462 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5196970105171204 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.249672070145607 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49817705154418945 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4595407247543335 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.46870675683021545 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6503087282180786 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2177303433418274 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06964285671710968 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.946601390838623 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 particle/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1756504625082016 + Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5625 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_2‚1 particle/accuracy_6_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04802677035331726 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010642197914421558 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02721637859940529 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4848484992980957 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17745612561702728 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12020960450172424 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03366174176335335 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9902740716934204 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6075867414474487 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01787603832781315 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1862524002790451 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9908159971237183 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4858555197715759 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27781441807746887 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47490403056144714 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11867816746234894 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1773967146873474 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5605034828186035 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5754227638244629 particle/accuracy_1_of_1‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4697107970714569 +jet/accuracy_3_of_5‚0.09599973261356354 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5546659231185913 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14965508878231049 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913790822029114 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 (loss/val_total_loss‚39.9488425724643 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907159805297852 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17259719967842102á™ÝR³0 c0myqu37j@ ÇâëÔÀÌñ0`Trainer.fit` stopped: `max_epochs=10` reached. ê »±ž´0 c0myqu37j‹ ÉâëÔð‘œizEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:22:58 • 0:00:00 0.90it/s v_num: qu370ðãÍ$µ0 c0myqu37j ÉâëÔЧ´i Z"¢4¶0/ _wandbruntime‚14594  _runtime‚14594Sû†%·0‚(’‚r